Multimodal Interface for Games: A Case Study with TinyML
Haoxuan Xie, Lam Chi Hou, Lap Tou Chau, Lei Weng, Xichen Wang, Yuxuan Guo, Giovanni Delnevo, Chiara Ceccarini, Chan–Tong Lam, Su-Kit Tang · 2024
Multimodal interfaces go beyond the traditional interaction through keyboard and mouse by incorporating multiple modes of interaction, such as touch, voice, gesture, and even gaze, to create more intuitive and immersive user experiences. This paper investigates how TinyML can be employed for multimodal interfaces in the context of games. An endless game in which the character has to avoid obstacles and fight monsters to advance has been developed. An Arduino Nano 33 BLE Sense is then used as the input device for the game by recognizing the hand gestures of the players.