Responsive audio feedback for hand gesture interaction to enhance immersion in audio-only games
Wenjie Wu, Stefan Rank · 2015
This thesis investigates different audio feedback designs for touch-less hand gesture interaction to enhance immersion and encourage intuitive user participation in audio-only environments. In particular, we aim to assess whether replacing explicit audio instructions for hand positions and movements with responsive audio feedback using environmental story-related audio cues that are suggestive of interaction methods leads to measurably higher immersion. The potential of using audio cues as feedback to improve the responsiveness of touch-less gesture interaction has been an active topic of study. We focus on methods for effectively informing players about gesture interaction methods through audio cues without breaking game immersion. As part of our research, we developed a framework for hand gesture detection based on the Leap Motion hardware and software. We also present a design approach for audio feedback that considers three stages, before, during and after a gesture sequence, and proposes corresponding design principles. In order to test different audio designs and support our hypothesis, two game versions using non-diegetic or diegetic audio feedback were created as testbeds for a between-subjects study. The study found that the version using environmental feedback led to higher immersion in terms of story and was consistently rated higher. In particular, compared to explicit vocal instructions, environmental story-related audio cues were found to be more helpful in guiding gesture interaction and in enhancing how real the game feels.