Multimodal language processing for mobile information access
Michael O. Johnston, Srinivas Bangalore, Amanda J. Stent, Gunaranjan Vasireddy, Patrick Ehlen · 2002
Interfaces for mobile information access need to allow users flexibility in their choice of modes and interaction style in accordance with their preferences, the task at hand, and their physical and social environment. This paper describes the approach to multimodal language processing in MATCH (Multimodal Access To City Help), a mobile multimodal speech-pen interface to restaurant and subway information for New York City. Finite-state methods for multimodal integration and understanding enable users to interact using pen, speech, or dynamic combinations of the two, and a speech-act based multimodal dialogue manager enables mixedinitiative multimodal dialogue. 1. LANGUAGE PROCESSING FOR MOBILE SYSTEMS Mobile information access devices (PDAs, tablet PCs, next generation phones) offer limited screen real estate and no keyboard or