Mind the Mix: Exploring the Cognitive Underpinnings of Multimodal Interaction in Augmented Reality Systems
May Jorella Lazaro, Sungho Kim · 2024
Exploring the intricate dynamics of Multimodal Interaction (MMI) in Augmented Reality (AR), this study presents a novel conceptual framework, crafted from a review of cognitive theories. Our research delves into how input modalities, output modalities, and their combinations uniquely influence user experiences in AR environments. Recognizing a gap in the existing MMI literature, especially within the AR context, we propose a conceptual framework to understand these complex relationships. Our framework pinpoints three critical factors: the choice of input modality, the verbal processing code of outputs, and the synergistic effects of input-output combinations. These elements are hypothesized to significantly impact user interaction and performance in AR systems. This work-in-progress not only contributes to the theoretical discourse in HCI but also sets the stage for future empirical investigations, aiming to enhance user-centered design in the evolving field of AR technology.