The area~ System: Exploring Real and Virtual Environments Through Gestural Ambisonics and Audio Augmented Reality
Sam Bilbow · 2021
The area~ System: Exploring Real and Virtual Environments Through Gestural Ambisonics and Audio Augmented Reality Sam Bilbow, University of Sussex In this paper, I outline the development and evaluation of the area~ system.area~ enables users to record, manipulate, and spatialise virtual audio samples or nodes around their immediate environment.Through a combination of ambisonics audio rendering and hand gesture tracking, this system calls attention to the ability of non-visual augmented reality (AR), here, audio augmented reality (AAR), to provide new aesthetic experiences of real and virtual environments.The system is contextualised within the move in computational art, and indeed, broader human computer interaction research, towards multisensory interaction.In particular, area~ is situated in the creative practice of works using multisensory AR as a medium to create expressive computational art.Through an autobiographical design study, these experiences are discussed in relation to the research question: "How can we better understand relationships between virtual and real environments through gesture and virtually placed audio augmented reality objects?"This hypothesis study proposes that new aesthetic experiences can result from the system and are waiting to be tested through user studies.The adoption of the Project North Star open-source AR head-mounted-display (HMD) could expand the possibilities of the area~ system by reducing the need to be tethered to a laptop and table for hand gesture input.In discussing the future development of the system and my research, I propose a devising practice-led method for creating and evaluating new multisensory AR (MSAR) Experiences; as well as the tantalising prospect of adding interaction between other sensory modalities to the area~ system, such as vision or smell, which would be made possible by the use of this open-source HMD.