Evaluation of a Multimodal Interaction System for Big Displays
Ana María Bernardos, Luca Bergesio, Juan Alberto Besada, José R. Casar · 2023
Big displays (ultrawalls) are increasingly present in professional, urban and leisure environments. Traditionally used as visualization tools, large-size screen surfaces may expand their interaction capabilities when integrated with sensors and mobile devices. This article describes a system that enables both voice and gesture control and mobile-mediated interaction with multiple information streams in a big display. The system offers support to a visualization model based on tiles and predefined layouts, which serves to handle the information streams and services. A set of specific commands combining voice and gesture control are preliminarily proposed. The vocabulary is enough to configure a command and monitoring service for fleets of drones. On this service scenario, the paper describes a study with ten users that has been carried out to verify that multimodal interaction can be efficient and perceived as acceptable, in particular when compared to mobile-enabled interaction. Results show that both proposals may deliver similar efficiency for simple tasks. Users are not reluctant to use multimodality, although they prefer voice over gestures and suggest combining both mobile and multimodal approaches.