MIBO – A Framework for the Integration of Multimodal Intuitive Controls in Smart Buildings
Sebastian Peters · 2016
With the Internet of Things expanding into our homes and offices, we can expect fixtures in buildings to become more interconnected and more numerous as part of an increasingly complex and powerful integrated smart environment. However, this raises new challenges in the area of usability since today's rooms are already cluttered with multiple user interfaces in the form of buttons and remote controls. In the future, as more technology is embedded into buildings, users must have the opportunity to choose and to combine a heterogeneous set of devices and modalities. Multimodal User Interaction technology aims at creating natural and intuitively usable interfaces, allowing a user to interact with systems in a way similar to human-to-human communication. The combination of modalities, such as gestures, speech and gaze-tracking provides occupants with an integrated and intuitive interface for diverse addressable fixtures in buildings. However, creating multimodal interfaces remains a complex and highly specialized task. This research describes MIBO – a framework to enable and define multimodal intuitive building controls. It supports the integration and combination of multiple interaction modalities, e.g., gesture recognition, speech recognition, eye-tracking but also traditional button controls and performs the multimodal fusion. It also provides a meta model and an extensible domain-specific language to describe a multimodal interaction model and to separate it from its implementation. This enables prototyping of new control systems for developers and allows end-users the configuration of desired interactions in a building. The framework has been applied in three real-world applications, each in at least two different buildings with more than 20 users, demonstrating the feasibility and efficiency of our approach.