Human-AI Handshaking: Supporting Extreme Sensemaking through Trustworthy Shared Perception

Abstract

Extreme sensemaking occurs when teams of people must build situational awareness in high-stakes, dynamic environments such as search and rescue, military, or security operations. These contexts are marked by uncertainty, fragmented information, and time-critical decisions that stretch human cognitive and physical limits. Artificial intelligence (AI) offers potential assistance, but distributed AI systems that rely on object detection, such as drone swarms, autonomous vehicles, and large-scale sensor networks, face their own challenges, including fluctuating accuracy, false detections, and fragile resilience under real-world dynamics. This dissertation introduces the "Human-AI Handshake", a novel human-AI interaction technique that incorporates human-in-the-loop (HITL) and crowd-in-the-loop (CITL) components to enhance object detection accuracy and shared perception in distributed systems. The Human-AI Handshake addresses core challenges by combining human feedback with AI model performance to mitigate uncertainties and improve trustworthiness. The foundation of the proposed concept comprises four key components: (1) evaluating HITL concepts for improving computer vision accuracy, (2) enhancing shared perception among distributed AI systems to enable better situational awareness, (3) ensuring AI trustworthiness through a synchronized perception trust framework, and (4) interpreting contextual awareness to help AI systems adapt to diverse, real-time scenarios. The technique is tested in extreme sensemaking applications, including augmented reality (AR)-assisted search and rescue, where accurate and reliable object detection is essential. Overall, the Human-AI Handshake provides an assured, scalable solution for improving AI assurance in distributed, dynamic environments, ensuring greater reliability, trust, and performance in critical operations.