Scholarly Works, Institute for Creativity, Arts, and Technology (ICAT)
Permanent URI for this collection
Research articles, presentations, and other scholarship
Browse
Recent Submissions
- A silent spring, or a new cacophony? Invasive plants as maestros of modern soundscapesBarney, Jacob N.; O'Malley, Grace; Ripa, Gabrielle N.; Drake, Joseph; Franusich, David; Mims, Meryl C. (Wiley, 2024-04-01)Sound plays a key role in ecosystem function and is a defining part of how humans experience nature. In the seminal book Silent Spring (Carson 1962), Rachel Carson warned of the ecological and environmental harm of pesticide usage by envisioning a future without birdsong. Soundscapes, or the acoustic patterns of a landscape through space and time, encompass both biological and physical processes (Pijanowski et al. 2011). Yet, they are often an underappreciated element of the natural world and the ways in which it is perceived. Scientists are only beginning to quantify changes to soundscapes, largely in response to anthropogenic sounds, but soundscape alteration is likely linked to many dimensions of global change. For example, invasive non-native species (hereafter, invasive species) are near-ubiquitous members of ecosystems globally and threaten both natural and managed ecosystems at great expense. Their impacts to soundscapes may be an important, yet largely unknown, threat to ecosystems and the human and economic systems they support.
- Interactive stories through robot musical theater for preschoolers’ STEAM educationChoi, Koeun; Yu, Shuqi; Kim, Jisun; Dong, Jia; Lee, Yeaji; Haines, Chelsea; Newbill, Phyllis; Upthegrove, Tanner; Wyatt, Ariana; Jeon, Myonghoon (2022)
- A child-robot musical theater afterschool program for promoting STEAM education: A case study and guidelinesDong, Jia; Choi, Koeun; Yu, Shuqi; Lee, Yeaji; Kim, Jisun; Vajir, Devanshu; Haines, Chelsea; Newbill, Phyllis; Wyatt, Ariana; Upthegrove, Tanner; Jeon, Myounghoon (Taylor & Francis, 2023-03-16)With the advancements of machine learning and AI technologies, robots have been more widely used in our everyday life and they have also been used in education. The present study introduces a 12-week child-robot theater afterschool program designed to promote science, technology, engineering, and mathematics (STEM) education with art elements (STEAM) for elementary students using social robots. Four modules were designed to introduce robot mechanisms as well as arts: Acting (anthropomorphism), Dance (robot movements), Music and Sounds (music composition), and Drawing (robot art). These modules provided children with basic knowledge about robotics and STEM and guided children to create a live robot theater play. A total of 16 students participated in the program, and 11 of them were involved in completing questionnaires and interviews regarding their perceptions towards robots, STEAM, and the afterschool program. Four afterschool program teachers participated in interviews, reflecting their perceptions of the program and observations of children’s experiences during the program. Our findings suggest that the present program effectively maintained children’s engagement and improved their interest in STEAM by connecting social robots and theater production. We conclude with design guidelines and recommendations for future research and programs.
- Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature ReviewJesso, Stephanie Tulk; Kelliher, Aisling; Sanghavi, Harsh; Martin, Thomas; Parker, Sarah H. (Frontiers, 2022-04-07)The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.
- Tapping into community expertise: stakeholder engagement in the design processMorshedzadeh, Elham; Dunkenberger, Mary Beth; Nagle, Lara; Ghasemi, Shiva; York, Laura; Horn, Kimberly (Taylor & Francis, 2022-10)The Connection to Care (C2C) project, a transdisciplinary work-in-progress, employs community-engaged participatory research and design methods at the nexus of policy adaptation and product innovations. C2C aims to advance practices that identify and leverage the critical junctures at which people with substance use disorder (SUD) seek lifesaving services and treatment, utilizing stakeholder input in all stages of design and development. Beginning in the Fall of 2018, members of our research team engaged with those at the forefront of the addiction crisis, including first responders, harm reduction and peer specialists, treatment providers, and individuals in recovery and in active substance use in a community greatly impacted by SUD. Through this engagement, the concept for programs and products representing a connection to care emerged, including the design of a backpack to meet the needs of individuals with SUD and those experiencing homelessness. From 2020 to 2022, more than 1,200 backpacks with lifesaving and self-care supplies have been distributed in local communities, as one component of the overall C2C initiative. The backpack is a recognized symbol of the program and has served as an impetus for further program and policy explorations, including as a lens to better understand the role of ongoing stigma. Though addiction science has evolved significantly in the wake of the opioid epidemic, artifacts of policies and practices that criminalize and stigmatize SUD remain as key challenges. This paper explains the steps that C2C has taken to address these challenges, and to empower a community that cares.
- Reimagining medical workspaces through on-site observations and bodystormingIshida, Aki; Martin, Thomas; Gracanin, Denis; Franusich, David; Buck, Carl; Parker, Sarah H.; Knapp, R. Benjamin; Haley, Vince; Zagarese, Vivian; Tasooji, Reza (2023-01)Clinicians in acute care hospitals experience highly stressful situations daily. They work long, variable hours, complete complex technical tasks, and must also be emotionally engaged with patients and families to meet the caring demands of this profession, which can lead to burnout. In response to these challenges, a multi-disciplinary team from Virginia Tech collaborated with Steelcase to study the impact of medical workspaces on the clinician experience and how those workspaces could be improved to reduce some of the sources of burnout. The team sought to identify conditions that could either aid or hinder clinician workflow and affect burnout rate, then based on interviews and in-situ ethnographic studies, generated design concepts for nurse stations, both centralized and mobile. Using digital and physical full-scale prototypes, we enacted clinical care scenarios to seek feedback and reflect on the design.
- Aegis Audio Engine: Integrating a Real-Time Analog Signal Processing, Pattern Recognition, and a Procedural Soundtrack in a Live Twelve-Perfomer Spectacle With Crowd ParticipationBukvic, Ivica Ico; Matthews, Michael (Georgia Institute of Technology, 2015-07)In the following paper we present Aegis: a procedural networked soundtrack engine driven by real-time analog signal analysis and pattern recognition. Aegis was originally conceived as part of Drummer Game, a game-performancespectacle hybrid research project focusing on the depiction of a battle portrayed using terracotta soldiers. In it, each of the twelve cohorts—divided into two armies of six—are led by a drummer-performer who issues commands by accurately drumming precomposed rhythmic patterns on an original Chinese war drum. The ensuing spectacle is envisioned to also accommodate large audience participation whose input determines the morale of the two armies. An analog signal analyzer utilizes efficient pattern recognition to decipher the desired action and feed it both into the game and the soundtrack engine. The soundtrack engine then uses this action, as well as messages from the gaming simulation, to determine the most appropriate soundtrack parameters while ensuring minimal repetition and seamless transitions between various clips that account for tempo, meter, and key changes. The ensuing simulation offers a comprehensive system for pattern-driven input, holistic situation assessment, and a soundtrack engine that aims to generate a seamless musical experience without having to resort to cross-fades and other simplistic transitions that tend to disrupt a soundtrack’s continuity.
- L2OrkMote: Reimagining a Low-Cost Wearable Controller for a Live Gesture-Centric Music PerformanceTsoukalas, Kyriakos D.; Kubalak, Joseph R.; Bukvic, Ivica Ico (ACM, 2018-06)Laptop orchestras create music, although digitally produced, in a collaborative live performance not unlike a traditional orchestra. The recent increase in interest and investment in this style of music creation has paved the way for novel methods for musicians to create and interact with music. To this end, a number of nontraditional instruments have been constructed that enable musicians to control sound production beyond pitch and volume, integrating filtering, musical effects, etc. Wii Remotes (WiiMotes) have seen heavy use in maker communities, including laptop orchestras, for their robust sensor array and low cost. The placement of sensors and the form factor of the device itself are suited for video games, not necessarily live music creation. In this paper, the authors present a new controller design, based on the WiiMote hardware platform, to address usability in gesture-centric music performance. Based on the pilot-study data, the new controller offers unrestricted two-hand gesture production, smaller footprint, and lower muscle strain.
- Introducing D⁴: An Interactive 3D Audio Rapid Prototyping and Transportable Rendering Environment Using High Density Loudspeaker ArraysBukvic, Ivica Ico (University of Michigan, 2016)With a growing number of multimedia venues and research spaces equipped with High Density Loudspeaker Arrays, there is a need for an integrative 3D audio spatialization system that offers both a scalable spatialization algorithm and a battery of supporting rapid prototyping tools for time-based editing, rendering, and interactive low-latency manipulation. D⁴ library aims to assist this newfound whitespace by introducing a Layer Based Amplitude Panning algorithm and a collection of rapid prototyping tools for the 3D time-based audio spatialization and data sonification. The ensuing ecosystem is designed to be transportable and scalable. It supports a broad array of configurations, from monophonic to as many as hardware can handle. D⁴’s rapid prototyping tools leverage oculocentric strategies to importing and spatially rendering multidimensional data and offer an array of new approaches to time-based spatial parameter manipulation and representation. The following paper presents unique affordances of D⁴’s rapid prototyping tools.
- Introducing a K-12 Mechatronic NIME KitTsoukalas, Kyriakos D.; Bukvic, Ivica Ico (ACM, 2018-06)The following paper introduces a new mechatronic NIME kit that uses new additions to the Pd-L2Ork visual programing environment and its K-12 learning module. It is designed to facilitate the creation of simple mechatronics systems for physical sound production in K- 12 and production scenarios. The new set of objects builds on the existing support for the Raspberry Pi platform to also include the use of electric actuators via the microcomputer’s GPIO system. Moreover, we discuss implications of the newly introduced kit in the creative and K-12 education scenarios by sharing observations from a series of pilot workshops, with particular focus on using mechatronic NIMEs as a catalyst for the development of programing skills.
- NIMEhub: Toward a Repository for Sharing and Archiving Instrument DesignsMcPherson, Andrew P.; Berdahl, Edgar; Lyons, Michael J.; Jensensius, Alexander Refsum; Bukvic, Ivica Ico; Knudson, Arve (ACM, 2016-07)This workshop will explore the potential creation of a community database of digital musical instrument (DMI) designs. In other research communities, reproducible research practices are common, including open-source software, open datasets, established evaluation methods and community standards for research practice. NIME could benefit from similar practices, both to share ideas amongst geographically distant researchers and to maintain instrument designs after their first performances. However, the needs of NIME are different from other communities on account of NIME's reliance on custom hardware designs and the interdependence of technology and arts practice. This half-day workshop will promote a community discussion of the potential benefits and challenges of a DMI repository and plan concrete steps toward its implementation.
- Introducing Locus: a NIME for Immersive Exocentric Aural EnvironmentsSardana, Disha; Joo, Woohun; Bukvic, Ivica Ico; Earle, Gregory D. (ACM, 2019-06)Locus is a NIME designed specifically for an interactive, immersive high density loudspeaker array environment. The system is based on a pointing mechanism to interact with a sound scene comprising 128 speakers. Users can point anywhere to interact with the system, and the spatial interaction utilizes motion capture, so it does not require a screen. Instead it is completely controlled via hand gestures using a glove that is populated with motion-tracking markers. The main purpose of this system is to offer intuitive physical interaction with the perimeter-based spatial sound sources. Further, its goal is to minimize user-worn technology and thereby enhance freedom of motion by utilizing environmental sensing devices, such as motion capture cameras or infrared sensors. The ensuing creativity enabling technology is applicable to a broad array of possible scenarios, from researching limits of human spatial hearing perception to facilitating learning and artistic performances, including dance. Below we describe our NIME design and implementation, its preliminary assessment, and offer a Unity-based toolkit to facilitate its broader deployment and adoption.
- 3D Time-Based Aural Data Representation Using D⁴ Library’s Layer Based Amplitude Panning AlgorithmBukvic, Ivica Ico (Georgia Institute of Technology, 2016-07)The following paper introduces a new Layer Based Amplitude Panning algorithm and supporting D⁴ library of rapid prototyping tools for the 3D time-based data representation using sound. The algorithm is designed to scale and support a broad array of configurations, with particular focus on High Density Loudspeaker Arrays (HDLAs). The supporting rapid prototyping tools are designed to leverage oculocentric strategies to importing, editing, and rendering data, offering an array of innovative approaches to spatial data editing and representation through the use of sound in HDLA scenarios. The ensuing D⁴ ecosystem aims to address the shortcomings of existing approaches to spatial aural representation of data, offers unique opportunities for furthering research in the spatial data audification and sonification, as well as transportable and scalable spatial media creation and production.
- Cinemacraft: Immersive Live Machinima as an Empathetic Musical Storytelling PlatformNarayanan, Siddhart; Bukvic, Ivica Ico (University of Michigan, 2016)In the following paper we present Cinemacraft, a technology-mediated immersive machinima platform for collaborative performance and musical human-computer interaction. To achieve this, Cinemacraft innovates upon a reverse-engineered version of Minecraft, offering a unique collection of live machinima production tools and a newly introduced Kinect HD module that allows for embodied interaction, including posture, arm movement, facial expressions, and a lip syncing based on captured voice input. The result is a malleable and accessible sensory fusion platform capable of delivering compelling live immersive and empathetic musical storytelling that through the use of low fidelity avatars also successfully sidesteps the uncanny valley.
- OPERAcraft: Blurring the Lines between Real and VirtualBukvic, Ivica Ico; Cahoon, Cody; Wyatt, Ariana; Cowden, Tracy; Dredger, Katie (University of Michigan, 2014-09)In the following paper we present an innovative approach to coupling gaming, telematics, machinima, and opera to produce a hybrid performance art form and an arts+technology education platform. To achieve this, we leverage a custom Minecraft video game and sandbox mod and pd-l2ork real-time digital signal processing environment. The result is a malleable telematic-ready platform capable of supporting a broad array of artistic forms beyond its original intent, including theatre, cinema, as well as machinima and other experimental genres.
- New Interfaces for Spatial Musical ExpressionBukvic, Ivica Ico; Sardana, Disha; Joo, Woohun (ACM, 2020-07)With the proliferation of venues equipped with the high den- sity loudspeaker arrays there is a growing interest in developing new interfaces for spatial musical expression (NISME). Of particular interest are interfaces that focus on the emancipation of the spatial domain as the primary dimension for musical expression. Here we present Monet NISME that leverages multitouch pressure-sensitive surface and the D⁴ library’s spatial mask and thereby allows for a unique approach to interactive spatialization. Further, we present a study with 22 participants designed to assess its usefulness and compare it to the Locus, a NISME introduced in 2019 as part of a localization study which is built on the same design principles of using natural gestural interaction with the spatial content. Lastly, we briefly discuss the utilization of both NISMEs in two artistic performances and propose a set of guidelines for further exploration in the NISME domain.
- Building Stem Career Interest Through Curriculum TreatmentsPeterson, Bryanne (Springer Open, 2020)Watson and McMahon’s (2005) work identified a need for research to examine the what and how of children’s career development learning; this research is a start to answering that call, specifically focusing on STEM career interest as a precursor to development due to the current needs nationally for an increase in the STEM pipeline. This study examined the impacts of design-based learning and scientific inquiry curriculum treatments with embedded career content on the career interest of fifth-grade students as compared to traditional classroom methods. Findings show an upward trend in interest with the use of these curriculum treatments, though the change is not significant in most career areas, likely due to the short time period of the unit and/or small n.
- Studies In Spatial Aural Perception: Establishing Foundations For Immersive SonificationBukvic, Ivica Ico; Earle, Gregory D.; Sardana, Disha; Joo, Woohun (Georgia Institute of Technology, 2019-06)The Spatial Audio Data Immersive Experience (SADIE) project aims to identify new foundational relationships pertaining to human spatial aural perception, and to validate existing relationships. Our infrastructure consists of an intuitive interaction interface, an immersive exocentric sonification environment, and a layer-based amplitude-panning algorithm. Here we highlight the system’s unique capabilities and provide findings from an initial externally funded study that focuses on the assessment of human aural spatial perception capacity. When compared to the existing body of literature focusing on egocentric spatial perception, our data show that an immersive exocentric environment enhances spatial perception, and that the physical implementation using high density loudspeaker arrays enables significantly improved spatial perception accuracy relative to the egocentric and virtual binaural approaches. The preliminary observations suggest that human spatial aural perception capacity in real-world-like immersive exocentric environments that allow for head and body movement is significantly greater than in egocentric scenarios where head and body movement is restricted. Therefore, in the design of immersive auditory displays, the use of immersive exocentric environments is advised. Further, our data identify a significant gap between physical and virtual human spatial aural perception accuracy, which suggests that further development of virtual aural immersion may be necessary before such an approach may be seen as a viable alternative.
- Reimagining Human Capacity For Location-Aware Aural Pattern Recognition: A Case For Immersive Exocentric SonificationBukvic, Ivica Ico; Earle, Gregory D. (Georgia Institute of Technology, 2018-06)The following paper presents a cross-disciplinary snapshot of 21st century research in sonification and leverages the review to identify a new immersive exocentric approach to studying human capacity to perceive spatial aural cues. The paper further defines immersive exocentric sonification, highlights its unique affordances, and presents an argument for its potential to fundamentally change the way we understand and study the human capacity for location-aware audio pattern recognition. Finally, the paper describes an example of an externally funded research project that aims to tackle this newfound research whitespace.
- #VTDITC vol. 9: Gender & Hip HopSmith, Blair Ebony; Jennings, Kyesha (Virginia Tech. University Libraries, 2018-04-19)Join Blair Ebony Smith [PhD Candidate at Syracuse University and artist-scholar with organizing collective Saving Our Lives Hear Our Truths (SOLHOT) as well as girl-band We Levitate], and Kyesha Jennings (Hip Hop Scholar and Curator/English Lecturer at NC State University) for a discussion of how issues of gender manifest in hip hop culture. Performances and an open cypher to follow. #VTDITC