An efficient unification-based multimodal language processor in multimodal input fusion
Yong Sun, Yu Shi, Fang Chen, Yuk Ying Chung · 2007
A Multimodal User Interface (MMUI) allows a user to interact with a computer in a way similar to human-to-human communication, for example, through speech and gesture. Being an essential component in MMUIs, Multimodal Input Fusion should be able to find the semantic interpretation of a user's intention from recognized multimodal symbols which are semantically complementary. We enhanced our efficient unification-based multimodal parsing processor, which has the potential to achieve low polynomial computational complexity while parsing versatile multimodal inputs within a speech and gesture based MMUI, to handle multimodal inputs from more than two modalities. Its ability to disambiguate speech recognition results with gesture recognition results was verified with an experiment. The analysis of experiment results demonstrates that the improvement is significant after applying this technique.