Visual display, pointing, and natural language
Antonella De Angeli, Walter Gerbino, Giulia Cassano, Daniela Petrelli · 1998
This paper examines user behavior during multimodal human-computer interaction (HCI). It discusses how pointing, natural language, and graphical layout should be integrated to enhance the usability of multimodal systems. Two experiments were run to study simulated systems capable of understanding written natural language and mouse-supported pointing gestures. Results allowed to: (a) develop a taxonomy of communication acts aimed at identifying targets; (b) determine the conditions under which specific referent identification strategies are likely to be produced; (c) suggest guidelines for designing effective multimodal interfaces; (d) show that performance is strongly influenced by interface graphical layout and by user expertise. Our study confirms the value of simulation as a tool for building HCI models and supports the basic idea that linguistic, visual, and motor cues can be integrated to favor effective multimodal communication.