Scaling Up Complex Sensemaking and Problem-Solving Through Crowdsourcing and Human-AI Collaboration

Abstract

Open Source Intelligence (OSINT) is widely used to collect and analyze publicly available information for applications such as countering disinformation, cybersecurity vulnerability assessment, journalism, law enforcement, and human rights investigations. However, scaling OSINT investigations remains difficult because analysts must process large volumes of unstructured and ephemeral data, verify uncertain information, operate diverse technical tools, and coordinate work across teams. These challenges are compounded by gaps in technical expertise and by the risks of introducing unreliable automation. In this dissertation, I investigate how such collaborative sensemaking and problem-solving tasks can be scaled through two complementary approaches: crowdsourcing to augment expert investigations and human--AI collaboration to support individuals and teams. First, I developed OSINT Research Studios, a flexible crowdsourcing framework in which trained novices support professional investigators through structured discovery and verification tasks under expert supervision. Second, I introduced the OSINT Clinic and conducted a longitudinal co-design study to identify the challenges novices face during collaborative vulnerability assessments and examine how generative AI can support planning, data collection, processing, analysis, dissemination, collaboration, and leadership. Third, I studied proactive generative AI agents in time-sensitive, co-located problem-solving by comparing facilitator and peer roles and measuring their effects on group performance, workload, communication, and coordination. Finally, I developed the OSINT Research Platform, a transparent and context-aware tool-calling LLM agent that can orchestrate OSINT tools, preserve human oversight, and provide next-step suggestions based on shared team activity. Overall, these studies contribute sociotechnical frameworks, empirical findings, and design requirements for scaling collaborative investigations. I show that effective scaling requires more than increasing participation or automating tasks; it requires systems that preserve context, verification, transparency, ethical boundaries, and human judgment. The dissertation offers design recommendations trustworthy human--AI collaboration with evolving roles of AI in collaborative work---from tool and personal assistant, to proactive teammate, to agentic collaborator.