Our Third Ear: A Multi-Sensory Experience of Sound

Abstract

Our Third Ear aims to create a multi-sensory experience by fusing sight, touch, and sound. By creating a means of physically feeling music, listeners can connect with songs, bands, and individual musicians on a profoundly personal level. The potential for unintended applications like learning to play an instrument, broadening the understanding of music for people with hearing impairments, or providing a means of therapy are also exciting prospects.

The purpose of this paper is to illustrate the process involved in creating a multi-sensory experience of music from concept to prototype. The culmination of interdisciplinary research and a broad range of creative technologies resulting in a working system. The multi-sensory experience consists of primarily tactile, but also visual responses triggered by music and executed in conjunction with aural music. Tactile investigation involved varied tactile sensations such as vibration, temperature, pressure, proprioception, and touch. Further research questioned the practicality, feasibility, and psychological impacts of using such sensations as well as where on the body such sensations would optimally be received. Visual research involved the visual representation of notes, chords, and sounds, as well as, how music could directly affect visuals in a real time environment. Additional research explored active interaction and passive interaction of visual cues using human computer interfaces.