ARGESTUREAID: A Voice-Based, Adaptive, and Context-Aware Conversational Assistant for Supporting Mid-Air Gesture Discovery and Execution
Anjali Khurana, Amy K. Karlson, Christopher M. Collins, Mengjie Yu, Vinay Jayaram, Hrvoje Benko, Parmit K. Chilana · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · 2026
End-users of immersive Augmented Reality (AR) applications struggle to discover and execute unfamiliar mid-air gestures. We introduce ARG esture A id , a novel proof-of-concept conversational assistant that provides adaptive, context-aware assistance tailored to users' tasks and gestural errors. By combining users' voice-based task descriptions with formalized representations of hand gesture data together with LLMs, ARG esture A id dynamically supports users through: (1) feedforward , directing users toward correct pose, position, and placement before initiating gestures, and (2) corrective feedback , offering targeted, adaptive instructions to address user errors during discovery and execution. Our initial feasibility study (N=15), which compared users' real-world interaction data against a manually labeled reference dataset, shows that ARG esture A id achieved an overall accuracy of 81.3%. Our follow-up user study (N=15) evaluated ARG esture A id against a BASELINE LLM-based voice assistant across unimanual and bimanual gestures. Results indicate that ARG esture A id achieved higher task completion rates and was rated more favorably for perceived discoverability and usefulness. These findings suggest that adaptive, in-context feedforward and corrective feedback, grounded in formal gesture representations and observed error patterns, may support users during mid-air gesture discovery and execution, and help them refine their understanding of required gestures. We discuss key considerations for designing future adaptive, context-aware AR systems in AR environments.