Demo: Seamless IoT Interaction in Mixed Reality Environments
Ryan Padamadan, Thilini Dinushika, Niruth Bogahawatta, Kanchana Thilakarathna · 2025
Mixed Reality systems for IoT control often rely on fragmented interfaces, cloud infrastructure, and lack spatial awareness. We present ViewIoT: a fully on-device framework that enables intuitive and spatially grounded interaction with nearby IoT devices using a head-mounted display. ViewIoT discovers and localizes devices by combining real-time 3D spatial meshes with lightweight wireless signal sensing, removing the need for manual setup or external infrastructure. At the core is a unified multimodal intent recognition pipeline, powered by an embedded language model, that interprets hand gestures, voice commands, gaze, and proximity signals to resolve user intent in under 100 milliseconds. Running on a Meta Quest 3S, our prototype supports real-time IoT control through gaze-based panels and natural interaction without relying on cloud services or pre-installed anchors. This work provides a practical foundation for seamless and infrastructure-free management of smart environments in a Mixed Reality setting.