Semantic Composition of Multimodal Actions in Constraint-based Grammars
Katya Saint-Amand, Alex Lascarides · Edinburgh Research Explorer (University of Edinburgh) · 2011
The use of speech-accompanying hand gestures to depict objects, to structure the discourse or to give directions is ubiquitous in face-to-face interaction. In this paper, we analyse multimodal signals consisting of speech and gesture from the perspective of semantic composition in constraint-based grammars: we elevate standard methods from linguistics to description of multimodal input so as to connect the semantics of the gestural signal to the semantics of the speech signal and to produce an integrated logical form.