Toward AI-Mediated Immersive Sensemaking with Gaze-Aware Semantic Interaction

Abstract

Motivation. Analysts who work with large text corpora must forage for evidence, con- nect disparate facts, and synthesize explanations, which imposes a heavy cognitive load. Immersive Analytics offers improving the experience with spatial memory and embodied interaction that can reduce this burden, but does not save the analyst from exhaustively browsing the corpus to find what matters. However, modern head-worn displays include eye tracking, creating an opportunity to infer an analyst's perceived interest implicitly and to provide timely, intelligible, attention-aware assistance that can essentially help in offloading some of the cognitive work. Problem. How can we model an analyst's interest from their gaze so that an AI assistant guides foraging and supports synthesis while preserving analysts' agency over the layout? Specifically, we need methods that (a) predict perceived relevance at document and term levels during multi-document investigations, and (b) expose those predictions through visual cues that help users make sense of complex evidence. Approach. We introduced a gaze-derived interest model that combines fixation duration and dwell count, adjusted for high-frequency terms, to compute GazeScore for documents and words. In parallel, we studied analysts' acceptance of different automation levels to ground design principles for a gaze-aware assistant. We operationalized the model in EyeST, an immersive analytic tool that presents two levels of visual cues to externalize the analyst's interest. Global cues provide interpretable, low-overhead signals by ranking and color-encoded evidence. Local cues reveal relationships between documents to promote discovery without clutter, while being grounded in the analyst's interest. We conducted a feasibility study that compared GazeScore to the analyst's perceived relevance. In parallel, we assessed analysts' acceptance of automated systems with a clustering task offering three levels of automation. The findings from the two studies enabled us to develop gaze-aware semantic interactions for immersive sensemaking, followed by two studies: one examining its effects on foraging, while the other tested the effects of adaptive annotation during synthesis. Results. GazeScore separated relevant from irrelevant content at the word level from the outset, enabling a real-time document relevance predictor with high precision. The clustering study showed that analysts favored assistance that preserves analyst's control over the layout and provides clear rationales. Subsequent studies revealed that global cues increased the efficiency of the analyst by helping them spend more time on relevant information while avoiding noise. Local cues encouraged individual exploration and surfaced overlooked but useful documents. Both gaze-derived cues guided analysts to essential clues vital to the sensemaking task, reduced perceived physical demand, and reduced the need for explicit externalization of the analyst's interest. Implications. The findings point toward a design pattern for AI-mediated immersive sense- making: invest in bootstrapping evidence, emphasize global high-level signals early, reveal local relationships on demand, and pair every suggestion with a clear rationale to preserve trust and agency. More broadly, this work extends semantic interaction into implicit chan- nels by showing how gaze can externalize evolving interest in real time. While our focus was on predicting perceived relevance, the approach opens pathways to incorporate other implicit signals to capture a richer picture of analysts' cognitive states. Together, these insights pave the way for more adaptive, trustworthy, and human-centered gaze-aware systems that deepen human–AI collaboration in immersive analytics.