Modeling and Interpreting Multimodal Inputs: A Semantic Integration Approach
Minh Tue Vo, Alex Waibel · 1997
Modem user interfaces can take advantage of multiple input modalities such as speech, gestures, handwriting.., to increase robustness and flexibility. The con-struction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammar structure for a broad class of multimodal applications. We also present a set of grammar-based Java tools that facilitate the construction of multimodal input processing modules, including a connectionist network for multimodal semantic integration.