Investigating the Use of Physiological and Behavioral Signals to Facilitate Empathic Human-AI Interaction for Daily Stress Management

Abstract

This dissertation explores the design and evaluation of Empathic Large Language Models (EmLLMs) for general mental health support. EmLLMs use physiological and behavioral signals to infer users' mental states (affective and cognitive) and accordingly generate empathic messages as adaptive interventions. Three core research goals guided this work: (1) systematically reviewing state-of-the-art methods for stress and affect recognition with physiological signals and for designing physiologically adaptive systems, (2) developing and evaluating physiology-driven EmLLM prototypes that integrate stress detection with LLM-based dialogue for stress intervention, and (3) evaluating the performance and stability of multimodal LLMs using behavioral signals for emotion recognition and supportive message generation. Findings from the systematic review highlight that physiological signals provide valuable insights into stress and affect, and that systems with physiology-driven adaptation are effective at improving both user experiences and mental health interventions. Autoethnographic and pilot studies with graduate students on different prototypes of physiology-driven EmLLMs demonstrate promise for daily stress management, and expert evaluations provide further insights into refining the design of physiology-driven EmLLMs for real-world and clinical use. Performance and stability evaluations of multimodal LLMs show that multimodal behavioral inputs, including voice and facial features, enhance emotion recognition and reasoning. However, model behavior varies across modalities, underscoring the need for robust evaluation, customization strategies, and protective safeguards for mental health applications. Overall, this dissertation offers a systematic review, empirical insights, and design guidelines for developing empathic, engaging, and effective digital mental health systems.