MNEMOSINE: A Modular Framework to Monitor Interaction in Multimodal Interactive Spaces
Fábio Barros, Bernardo Marques, António Teixeira, Samuel Silva · IEEE Access · 2025
The evolution of human-machine interfaces has resulted in the proposal of increasingly complex multimodal solutions, encompassing a diverse set of input and output modalities, towards more natural and efficient interaction paradigms. Supporting the required range of interaction possibilities entails the development of complex systems which, as the number of features and involved technologies grows, pose challenges to research and development. This arises from the inherent difficulty in understanding how the system is working and how specific modules are behaving, important during development, integration, and testing, but also during deployment of these systems for evaluation by users, whether in a lab or in real environments. For these scenarios, approaches based on conventional logging or post-task questionnaires are no longer viable. In response, a first stage to tackle these challenges is proposed based on a systematic approach to monitoring interaction in these multimodal architectures. To this end, this work presents and illustrates the use of MNEMOSINE, the first instantiation of a framework offering real-time monitoring capabilities and event storage mechanisms for multimodal architectures. MNEMOSINE can serve as grounds for developers and researchers to get a structured view over current system status and creates the conditions supporting the analysis of system behavior over time. This aims to enhance the understanding and supervision of interactions within multimodal systems and can, in turn, foster improved system development and refinement, subsequently supporting the integration of increasingly multimodal systems into daily life.