Exploring the Usability of Non-verbal Vocal Interaction (NVVI) and a Pitch Based Implementation

Natural user interfaces, including verbal vocal interactions like speech processing, are ubiquitous and commonly used in both industry and academic settings. However, this field is limited by the speech and language components. Non-verbal vocal interaction (NVVI) provides further opportunities for people to use their vocals as an input modality. Despite the many possibilities of NVVI input modalities, such as whistling, humming, and tongue clicking, the field is niche and literature is few and far between. This work attempts to address these gaps, as well as the small sample sizes of performed studies of prior work. The problem definition is defined as to perform a large-scale study exploring a pitch-based NVVI modality that uses a relative pitch interaction technique to offer a continuous mode of one-dimensional interaction. A user study is outlined and performed via an ecosystem comprising of Amazon Mechanical Turk for recruitment and study access, a modularized study website, and a secure server that stores the study results, tasks users with controlling a slider with the NVVI technique by humming and whistling, in addition to using the computer mouse to perform these tasks as a baseline. In total, 72 participants' results are considered for analysis. Results show that the pitch based NVVI technique used in this study does not follow Fitts' Law, is not as performant as the computer mouse, humming is a more performant modality with the NVVI technique than whistling, and that participants experienced a significantly higher task workload using the NVVI technique than the computer mouse. Using the results of this study and from reviewed literature, an NVVI framework is developed and implemented as a contribution of this work.