Scholarly Works, Institute for Creativity, Arts, and Technology (ICAT)

Permanent URI for this collection

https://hdl.handle.net/10919/24352

Research articles, presentations, and other scholarship

Browse

Now showing 1 - 20 of 39

3D Time-Based Aural Data Representation Using D⁴ Library’s Layer Based Amplitude Panning Algorithm
Bukvic, Ivica Ico (Georgia Institute of Technology, 2016-07)
The following paper introduces a new Layer Based Amplitude Panning algorithm and supporting D⁴ library of rapid prototyping tools for the 3D time-based data representation using sound. The algorithm is designed to scale and support a broad array of configurations, with particular focus on High Density Loudspeaker Arrays (HDLAs). The supporting rapid prototyping tools are designed to leverage oculocentric strategies to importing, editing, and rendering data, offering an array of innovative approaches to spatial data editing and representation through the use of sound in HDLA scenarios. The ensuing D⁴ ecosystem aims to address the shortcomings of existing approaches to spatial aural representation of data, offers unique opportunities for furthering research in the spatial data audification and sonification, as well as transportable and scalable spatial media creation and production.
Aegis Audio Engine: Integrating a Real-Time Analog Signal Processing, Pattern Recognition, and a Procedural Soundtrack in a Live Twelve-Perfomer Spectacle With Crowd Participation
Bukvic, Ivica Ico; Matthews, Michael (Georgia Institute of Technology, 2015-07)
In the following paper we present Aegis: a procedural networked soundtrack engine driven by real-time analog signal analysis and pattern recognition. Aegis was originally conceived as part of Drummer Game, a game-performancespectacle hybrid research project focusing on the depiction of a battle portrayed using terracotta soldiers. In it, each of the twelve cohorts—divided into two armies of six—are led by a drummer-performer who issues commands by accurately drumming precomposed rhythmic patterns on an original Chinese war drum. The ensuing spectacle is envisioned to also accommodate large audience participation whose input determines the morale of the two armies. An analog signal analyzer utilizes efficient pattern recognition to decipher the desired action and feed it both into the game and the soundtrack engine. The soundtrack engine then uses this action, as well as messages from the gaming simulation, to determine the most appropriate soundtrack parameters while ensuring minimal repetition and seamless transitions between various clips that account for tempo, meter, and key changes. The ensuing simulation offers a comprehensive system for pattern-driven input, holistic situation assessment, and a soundtrack engine that aims to generate a seamless musical experience without having to resort to cross-fades and other simplistic transitions that tend to disrupt a soundtrack’s continuity.
AffecTech-an affect-aware interactive AV Artwork
Coghlan, Niall; Jaimovich, Javier; Knapp, R. Benjamin; O’Brien, Donal; Ortiz, Miguel A. (ISEA International, 2009)
New developments in real-time computing and body-worn sensor technology allow us to explore not just visible gestures using inertial sensors, but also invisible changes in an individual’s physiological state using bio-sensors (Kim & André 2008). This creates an opportunity for a more intimate interaction between the observer and technology-based art (Gonsalves 2008). We present a technical overview of the AffecTech system; a bio-signal based interactive audiovisual installation commissioned as part of the pre-ISEA symposium in November 2008. Observers were invited to sit on one of 2 sensor-enhanced chairs (Coghlan & Knapp 2008), which transmitted physiological data about the occupant to a central control system. This data was used to control and modulate interactive visuals, live video feeds and a surround sound score, with events and interactions dependent on the observers’ affective/emotional state and the disparity or similarity between the bio-signals of the chairs occupants. This technical overview is followed by an examination of the outcomes of the project, from both the artistic and technical viewpoints, with recommendations for modification in future implementations.
Affective Feedback in a Virtual Reality based Intelligent Supermarket
Saha, Deba Pratim; Martin, Thomas L.; Knapp, R. Benjamin (ACM, 2017)
The probabilistic nature of the inferences in a context-aware intelligent environment (CAIE) renders them vulnerable to erroneous decisions resulting in wrong services. Learning to recognize a user’s negative reactions to such wrong services will enable a CAIE to anticipate a service’s appropriateness. We propose a framework for continuous measurement of physiology to infer a user’s negative-emotions arising from receiving wrong services, thereby implementing an implicit-feedback loop in the CAIE system. To induce such negative-emotions, in this paper, we present a virtualreality (VR) based experimental platform while collecting real-time physiological data from ambulatory wearable sensors. Results from the electrodermal activity (EDA) data analysis reveal patterns that correlate with known features of negative-emotions, indicating the possibility to infer service appropriateness from user’s reactions to a service, thereby closing an implicit-feedback loop for the CAIE.
Biosignal-driven Art: Beyond biofeedback
Ortiz, Miguel A.; Coghlan, Niall; Jaimovich, Javier; Knapp, R. Benjamin (CMMAS, 2011)
Biosignal monitoring in interactive arts, although present for over forty years, remains a relatively little known field of research within the artistic community as compared to other sensing technologies. Since the early 1960s, an ever-increasing number of artists have collaborated with neuroscientists, physicians and electrical engineers, in order to devise means that allow for the acquisition of the minuscule electrical potentials generated by the human body. This has enabled direct manifestations of human physiology to be incorporated into interactive artworks. This paper presents an introduction to this field of artistic practice and scientific research that uses human physiology as its main element. A brief introduction to the main concepts and history of biosignal-driven art is followed by a review of various artworks and scientific enquiry developed by the authors. This aims at giving a complete overview of the various strategies developed for biosignal-driven interactive art.
Building Stem Career Interest Through Curriculum Treatments
Peterson, Bryanne (Springer Open, 2020)
Watson and McMahon’s (2005) work identified a need for research to examine the what and how of children’s career development learning; this research is a start to answering that call, specifically focusing on STEM career interest as a precursor to development due to the current needs nationally for an increase in the STEM pipeline. This study examined the impacts of design-based learning and scientific inquiry curriculum treatments with embedded career content on the career interest of fifth-grade students as compared to traditional classroom methods. Findings show an upward trend in interest with the use of these curriculum treatments, though the change is not significant in most career areas, likely due to the short time period of the unit and/or small n.
A child-robot musical theater afterschool program for promoting STEAM education: A case study and guidelines
Dong, Jia; Choi, Koeun; Yu, Shuqi; Lee, Yeaji; Kim, Jisun; Vajir, Devanshu; Haines, Chelsea; Newbill, Phyllis; Wyatt, Ariana; Upthegrove, Tanner; Jeon, Myounghoon (Taylor & Francis, 2023-03-16)
With the advancements of machine learning and AI technologies, robots have been more widely used in our everyday life and they have also been used in education. The present study introduces a 12-week child-robot theater afterschool program designed to promote science, technology, engineering, and mathematics (STEM) education with art elements (STEAM) for elementary students using social robots. Four modules were designed to introduce robot mechanisms as well as arts: Acting (anthropomorphism), Dance (robot movements), Music and Sounds (music composition), and Drawing (robot art). These modules provided children with basic knowledge about robotics and STEM and guided children to create a live robot theater play. A total of 16 students participated in the program, and 11 of them were involved in completing questionnaires and interviews regarding their perceptions towards robots, STEAM, and the afterschool program. Four afterschool program teachers participated in interviews, reflecting their perceptions of the program and observations of children’s experiences during the program. Our findings suggest that the present program effectively maintained children’s engagement and improved their interest in STEAM by connecting social robots and theater production. We conclude with design guidelines and recommendations for future research and programs.
Cinemacraft: Immersive Live Machinima as an Empathetic Musical Storytelling Platform
Narayanan, Siddhart; Bukvic, Ivica Ico (University of Michigan, 2016)
In the following paper we present Cinemacraft, a technology-mediated immersive machinima platform for collaborative performance and musical human-computer interaction. To achieve this, Cinemacraft innovates upon a reverse-engineered version of Minecraft, offering a unique collection of live machinima production tools and a newly introduced Kinect HD module that allows for embodied interaction, including posture, arm movement, facial expressions, and a lip syncing based on captured voice input. The result is a malleable and accessible sensory fusion platform capable of delivering compelling live immersive and empathetic musical storytelling that through the use of low fidelity avatars also successfully sidesteps the uncanny valley.
Contagion of Physiological Correlates of Emotion between Performer and Audience: An Exploratory Study
Jaimovich, Javier; Coghlan, Niall; Knapp, R. Benjamin (2010)
Musical and performance experiences are often described as evoking powerful emotions, both in the listener/observer and player/performer. There is a significant body of literature describing these experiences along with related work examining physiological changes in the body during music listening and the physiological correlates of emotional state. However there are still open questions as to how and why, emotional responses may be triggered by a performance, how audiences may be influenced by a performers mental or emotional state and what effect the presence of an audience has on performers. We present a pilot study and some initial findings of our investigations into these questions, utilising a custom software and hardware system we have developed. Although this research is still at a pilot stage, our initial experiments point towards significant correlation between the physiological states of performers and audiences and we here present the system, the experiments and our preliminary data.
Creating a Network of Integral Music Controllers
Knapp, R. Benjamin; Cook, Perry R. (NIME, 2006)
In this paper, we describe the networking of multiple Integral Music Controllers (IMCs) to enable an entirely new method for creating music by tapping into the composite gestures and emotions of not just one, but many performers. The concept and operation of an IMC is reviewed as well as its use in a network of IMC controllers. We then introduce a new technique of Integral Music Control by assessing the composite gesture(s) and emotion(s) of a group of performers through the use of a wireless mesh network. The Telemuse, an IMC designed precisely for this kind of performance, is described and its use in a new musical performance project under development by the authors is discussed.
Creating Biosignal Algorithms for Musical Applications from an Extensive Physiological Database
Jaimovich, Javier; Knapp, R. Benjamin (NIME, 2015)
Previously the design of algorithms and parameter calibration for biosignal music performances has been based on testing with a small number of individuals - in fact usually the performer themselves. This paper uses the data collected from over 4000 people to begin to create a truly robust set of algorithms for heart rate and electrodermal activity measures, as well as the understanding of how the calibration of these vary by individual.
Design of a Mobile Brain Training Tool for Seniors: Motivational Requirements
O’Brien, Donal; Knapp, R. Benjamin; Thompson, Oonagh; Craig, David; Barrett, Suzanne (ICST, 2012)
The overall goal of this project is to design, develop, and validate a mobile phone-based ‘brain training’ software suite targeted at senior users, using iterative person-centered design methodologies, that will permit a subsequent clinical trial of cognitive stimulation efficacy known as theSONIC²S Study (Stirling-Oregon-Northern Ireland-Chicago Cognitive Stimulation Study). The SONIC²S Study will represent a long term (c. 15 year), very large scale (n=12,000), embedded clinical trial that aims to definitively establish whether or not brain training acts to prevent dementia or cognitive decline. It is anticipated that participant compliance in such a study will be a significant concern. This study reports on a series of focus groups, usability studies and field trials which sought to identify the key motivational factors influencing seniors’ engagement with mobile brain-training technology in order to inform the design of a bespoke tool which is acceptable/enjoyable to target users.
Disciplinary Influences on the Professional Identity of Civil Engineering Students: Starting the Conversation
Groen, Cassandra J.; Simmons, Denise Rutledge; McNair, Elizabeth D. (2016-06)
As the discipline of civil engineering has evolved from an apprentice-based trade to a socially-engaged profession, the role of the civil engineer has responded to shifts within the ever-changing culture of society. These shifts and historical events have directly influenced what is considered to be valued civil engineering knowledge, behaviors, and practices that we teach to students during their undergraduate careers. As part of a larger grounded theory study that is currently being conducted by the authors, the purpose of this paper is two-fold. First, we present the topic of professional identity formation as heavily influenced by unique historical events that shape the civil engineering discipline. . To establish the connection between identity formation and the history of civil engineering, we interpret historical events as constituents that create a disciplinary identity that is communicated to and subjectively applied by students during their undergraduate careers. Second, we hope to promote and invoke conversations surrounding the relevancy of civil engineering professional identity formation in engineering education among our colleagues within the technical disciplines. Through this paper, we add to ongoing research exploring the professional formation of engineering identities and promote discussions surround this topic at the disciplinary level. While most research conducted on identity formation has been generalized to include all or most engineering disciplines, we focus our discussion solely on professional identity formation within the civil engineering discipline. To reinforce the relationship between the history of the civil engineering profession and students’ professional identity formation, we review the literature on these two areas of inquiry. In particular, we will frame our paper using the following key discussion points: 1) providing a brief overview of key historical events of civil engineering in the United States; 2) discussing the influence of this history on instructor pedagogies and student learning within civil engineering education; and 3) conceptualizing this learning process as a means of professional identity formation. From this work, we will begin to understand how major historical shifts within our discipline maintain the potential to impact its future as we educate the next generation of civil engineering students. To conclude this paper, we will introduce current research that is being conducted by the authors to further understand the nuances of professional identity formation in undergraduate civil engineering students and how instructors may help or hinder that development.
The Emotion in Motion Experiment: Using an Interactive Installation as a Means for Understanding Emotional Response to Music
Jaimovich, Javier; Ortiz, Miguel A.; Coghlan, Niall; Knapp, R. Benjamin (NIME, 2012)
In order to further understand our emotional reaction to music, a museum-based installation was designed to collect physiological and self-report data from people listening to music. This demo will describe the technical implementation of this installation as a tool for collecting large samples of data in public spaces. The Emotion in Motion terminal is built upon a standard desktop computer running Max/MSP and using sensors that measure physiological indicators of emotion that are connected to an Arduino. The terminal has been installed in museums and galleries in Europe and the USA, helping create the largest database of physiology and self-report data while listening to music.
Emotion in Motion: A Reimagined Framework for Biomusical/Emotional Interaction
Bortz, Brennon; Jaimovich, Javier; Knapp, R. Benjamin (NIME, 2015)
Over the past four years Emotion in Motion, a long running experiment, has amassed the world’s largest database of human physiology associated with emotion in response to the presentation of various selections of musical works. What began as a doctoral research study of participants in Dublin, Ireland, and New York City has grown to include over ten thousand emotional responses to musical experiences from participants across the world, from new installations in Norway, Singapore, the Philippines, and Taiwan. The most recent iteration of Emotion in Motion is currently underway in Taipei City, Taiwan. Preparation for this installation gave the authors an opportunity to reimagine the architecture of Emotion in Motion, allowing for a wider range of potential applications than were originally possible with the initial development of the tools that drive the experiment. Now more than an experiment, Emotion in Motion is a framework for developing myriad emotional/ musical/biomusical interactions with co-located or remote participants. This paper describes the development of this flexible, open-source framework and includes discussion of its various components: hardware agnostic sensor inputs, refined physiological signal processing tools, a public database of data collected during various instantiations of applications built on the framework, and the web application frontend and backend. We also discuss our ongoing work with this tool, and provide the reader with other potential applications that they might realize in using Emotion in Motion.
Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review
Jesso, Stephanie Tulk; Kelliher, Aisling; Sanghavi, Harsh; Martin, Thomas; Parker, Sarah H. (Frontiers, 2022-04-07)
The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.
Inner-Active Art: An examination of aesthetic and mapping issues in physiologically based artworks
Coghlan, Niall; Knapp, R. Benjamin (ISEA, 2009)
Much art seeks to describe or stimulate the feelings and emotions of the viewer, through both abstract and literal representation. With the exponential increase in computing power over recent years we also seek new ways of interacting with technology and exploring the virtual world. Physiological signals from the human body provide us with a view into the autonomic nervous system, that part of the nervous system largely unmediated by the direct intentions of the viewer. With the appropriate choice of signals and processing, we can even develop systems with the ability to interact with us on an emotional level - machines that know how we feel and can react accordingly (Haag et. al., 2004). This gives us the ability to see into and map the interior worlds of artists and viewers through a direct and visceral connection, the human body itself. A key issue in the development of physiologically based artwork is to make the observer-artwork dialogue meaningful to the observer, a question of translating the input bio-signals to visual, auditory or experiential events. We have yet to develop a suitable language for this dialogue and so this paper seeks to explore some potential mappings for bio-signal art, illustrated using several case studies from past and current works (Knapp et.al., 2008) (Gonsalves, 2009). We also examine some of the other philosophical and artistic issues involved in 'affective' and bio-art such as monitoring emotion v. engendering emotion, the involvement of the observer in creating and contributing to bio-signal art and strategies for effectively developing such works.
The integral Music controller: Introducing a Direct Emotional Interface to gestural control of sound synthesis
Knapp, R. Benjamin; Cook, Perry R. (Michigan Publishing, 2005-09)
This paper describes the concept of the integral music controller (IMC), a controller that combines gestural interface with direct emotional control of a digital musical instrument. This new controller enables the performer to move smoothly between direct physical interaction with an acoustical musical instrument, and gestural/emotional control of the instrument’s physical model. The use of physiological signals to determine gesture and emotion is an important component of the IMC. The design of a wireless IMC using physiological signals is described and possible mappings to sound synthesis parameters are explored. Controlling higher level musical systems such as conducting and style modelling is also proposed.
Interactive stories through robot musical theater for preschoolers’ STEAM education
Choi, Koeun; Yu, Shuqi; Kim, Jisun; Dong, Jia; Lee, Yeaji; Haines, Chelsea; Newbill, Phyllis; Upthegrove, Tanner; Wyatt, Ariana; Jeon, Myounghoon (2022)
Introducing a K-12 Mechatronic NIME Kit
Tsoukalas, Kyriakos D.; Bukvic, Ivica Ico (ACM, 2018-06)
The following paper introduces a new mechatronic NIME kit that uses new additions to the Pd-L2Ork visual programing environment and its K-12 learning module. It is designed to facilitate the creation of simple mechatronics systems for physical sound production in K- 12 and production scenarios. The new set of objects builds on the existing support for the Raspberry Pi platform to also include the use of electric actuators via the microcomputer’s GPIO system. Moreover, we discuss implications of the newly introduced kit in the creative and K-12 education scenarios by sharing observations from a series of pilot workshops, with particular focus on using mechatronic NIMEs as a catalyst for the development of programing skills.

Browse

Browsing Scholarly Works, Institute for Creativity, Arts, and Technology (ICAT) by Title

Results Per Page

Sort Options