Improving the performance of speech-gesture multimodal interface in non-ideal environments
Fiolisya Faustine Ambadar, Jude Joseph Lamug Martinez · Procedia Computer Science · 2023
Multimodal interfaces have enhanced human-computer interaction by enabling users to interact with computers using a combination of multiple input modes, providing increased accessibility to a wider range of users in various situations. The multimodal system's ability to process multiple input modes allows it to rely on one input modal given that the second modal is unable to function due to exposure to extreme environments. This study will analyse a speech-gesture multimodal interface framework and the prototype that was initially developed by Sindy Dewanti and have been improved upon by Regita Isada. To further improve the framework and prototype's performance, this study will evaluate and resolve the issues encountered in the previous study regarding the configuration of each modal's confidence levels, environment detection, weight calculation, and how the unification process selects a final semantic. Upon implementing the changes, the prototype was tested under three environmental conditions: normal, moderate, and extreme in both unimodal and multimodal mode. The test results show that the prototype was able to deliver the expected results with improved accuracy in multimodal mode as compared to the previous study. Nonetheless, the way that the modals perform, and the unification process can still be further improved