Improving Response Time through Multimodal Integration Pattern Modeling
Roman Hák, Tomáš Zeman · 2016
While researchers have focused primarily on accuracy when addressing multimodal input segmentation, response time (or latency) has been rather overlooked in their work, despite its unquestionable importance. We propose a method of the input segmentation through integration pattern modeling that provides a significant improvement in response time over the state-of-the-art approaches, while maintaining remarkably high accuracy (98-99%). To this end, a new Bayesian Belief Network classification model was designed based on the recent empirical evidence about users' multimodal integration patterns. The model is employed in a procedure to segment related inputs into multimodal units. Using the introduced procedure the response time can be improved to 0.8 seconds for sequential integrators and even dropped bellow 0.5 s for simultaneous, which represents a relative improvement of 20% and 50%, resp., at the very least. Although demonstrated on a combination of speech and gestures, the suggested approach can be generalized to a broad range of other modality mixtures.