Multimodal generation, spatial language and illustration
Luis A. Pineda · 2003
In this paper the role of graphical illustration in multimodal presentation systems is discussed. First, a review of two current approaches to multimodal generation systems is presented; from this, the question how illustration supports the interpretation process is addressed from both a perceptual and a cognitive perspective. Following Jackendoff's semantics, it is described how representations and computational processes involving the identification of objects, on the one hand, and those involving the interpretation of spatial relations, like position, size and orientation, on the other, have very different cognitive and computational properties. The relation between the meaning of an expression and its interpretation in relation to a multimodal context is also discussed. Finally, on the basis of the properties of the language that is used to talk about space, and the contextual factors of the interpretation of multimodal expressions, a new strategy for the design of multimodal presentation systems is presented.