Designing Physiological Connections in Artificial Intelligence: Towards integration of an AI collaborator into a musical ensemble for group aesthetics and control using biofeedback with affective states

Abstract

We are artistic and creative human beings. As expert systems encroach on creative or artistic processes, rather than vilify or create certainty, how can we co-create with AI as an agent and have some synergy for the future? While some systems move towards excluding human interaction in favor of AI-generated processes, there is also a growing space for more physiological human-computer interactions. Artificial intelligence in music is often evalu- ated through autonomous generation, leaving unresolved how computational systems can support collaboration while preserving human agency. This dissertation develops Affective Computing in Musical Collaboration (ACMC), an interdisciplinary framework that treats affect as expressed, sensed, reported, negotiated, and acted upon within musical interaction rather than only as a classification label. Informed by participatory and transition-design principles, the project integrated a Russell circumplex interface, low-cost MAX30102 heart- rate and blood-oxygen sensing, Grove skin-conductance sensing, Pd-L2Ork/L2Ork Tweeter, Python, Open Sound Control, event logging, statistical analysis, and machine learning. A related symbolic-generation study using a human-classified emotional MIDI dataset found that continuous-concatenated transformer conditioning performed best among the evaluated conditioning methods. The empirical work followed a validation-to-interaction sequence. Phase 1 involved eight participants and 80 passive-listening trials across ten high-arousal, mixed-valence excerpts. All trials were reconstructed, heart-rate change showed a reproducible association with target valence, and self-reported valence retained modest stimulus ordering. However, baseline-relative directional concordance ranged from 33.8% to 47.5%, below the prespecified 80% progression criterion. Phase 2 was therefore interpreted as an exploratory feasibility study. Fifty participants completed 600 interactive Solo, Transition, and Drop trials. Skin conductance increased during and after interaction, heart rate rose most consistently after events, and SpO2 showed a modest post-event decrease. Drop examples were rated more arousing than Solo and Transition examples, whereas valence varied more by individual example than by musical function. Complete-case System Usability Scale scores averaged 81.80/100, and post-task and qualitative results indicated strong usability, enjoyment, learning, and creative exploration. The findings establish the feasibility and expressive value of an open-source, psychophysiologically responsive musical interface, but not a universal emotion decoder. Physiological and audio-derived affect estimates agreed weakly with independent retrospective ratings; participant-grouped models produced negative held-out R2 values, and function classification remained near chance. The planned Biopac reference-device comparison was not completed, so medical-grade sensor accuracy is not claimed. Ensemble consensus and audience coupling were also not tested. The dissertation contributes the ACMC framework, an open research stack, a validation model separating acquisition, physiological sensitivity, agreement, prediction, usability, and collaboration, and a design direction for personalized, transparent, mixed-initiative musical AI that treats physiology as contextual evidence rather than unquestioned authority.