A Framework of Freehand Gesture Interaction: Techniques, Guidelines, and Applications

Freehand gestures have long been considered to potentially deliver natural, intuitive, terse but powerful human-computer interaction techniques. Over years, researchers have been attempting to employ freehand gestures as an alternative input modality to the conventional devices (e.g. keyboard and mouse) in a wide array of application domains, and a huge number of gesture recognition systems and gesture-based interaction techniques have been created in lab. However, a fundamental question remains: is it possible to establish an interaction framework so that we may approach freehand gestural interaction from a systematic perspective, and design coherent and consistent freehand gesture-based human-computer interaction experience? Existing research tends to focus on the technologies that enable the gestural interaction, or on the novel design of gestural interaction techniques for specific tasks and applications. Such "point designs" are claimed to be insufficient, and an existing application-specific design lends very limited insights and guidance to design problems in another application. An interaction framework allows us to move from individual designs to a more holistic approach.

The goal of this research is to construct a framework to support a systematic approach for designing freehand gesture-based interactions. Toward this goal our research began with a review and examination of the gesture interaction literature, followed by an analysis of the essential components of an interaction framework. We then proposed and justified the scope of research and the approach we took to construct the interaction framework. We have designed and evaluated (analytically and empirically) gestural interaction techniques for two broad categories of freehand gestures we specified — spatial gestures, and surface gestures. In the design activity, we have discovered and proposed the core design principles and guidelines, and validated them via user studies. Finally, we assessed the ability of the freehand gesture interaction framework we have constructed to help designers create new applications and designs, by putting together a few proof-of-concept examples of a coherent and consistent freehand gesture user interface.