Cross-Modal Interaction in Graphical Communication
Ichiro Umata, Yasuhiro Katagiri · eScholarship (California Digital Library) · 2004
Cross-modal interaction in graphical communication is observed in collaborative problem-solving settings.Graphical communications, such as dialogues using maps, drawings, or pictures, provide people with two independent modalities: speech and drawing.Although the amount of drawing/self-speech overlap is strongly affected by activity-dependent constraints imposed by the task, the amount of drawing/partner's speech overlap is affected only weakly by these constraints.However, they do affect the function of the utterances in the case of drawing/partner's speech overlap.These results show that activity-level constraints affect the way speech coordinates drawing activities in cross-modal interaction.Furthermore, it suggests that turn-taking in multimodal communication requires general analyses integrating the functions of different modalities.