On what happens in speech and gesture when communication is unsuccessful - eScholarship
Marieke Hoetjes, Emiel Krahmer, Marc G. J. Swerts · Proceedings of the Annual Meeting of the Cognitive Science Society · 2014
On What Happens in Speech and Gesture when Communication is Unsuccessful Marieke Hoetjes ([email protected]) Emiel Krahmer ([email protected]) Marc Swerts ([email protected]) Tilburg center for Cognition and Communication (TiCC), Tilburg University, the Netherlands. Abstract Repeated references have been found to be reduced as compared to references that are not repeated, both in speech and in gesture. In the present study we wanted to see whether certain factors can inhibit this reduction in repeated references. In a production experiment, speakers were confronted with negative feedback after an initial description of an object, indicating that the communication was unsuccessful. We found that after initial negative feedback, second references were reduced with regard to all speech variables. When the speakers were confronted with additional negative feedback, the ensuing third references were increased in the number of words and the duration, as compared to the second references, but further reduced in their speech rate. Gesture rate increased in third references as compared to initial references. After (repeated) instances of unsuccessful communication, speakers speak slower and increase their gesture rate, thereby making their repeated references clearer for the addressee. Keywords: Gesture; repeated references; negative feedback. Introduction Repeated references in speech and gesture Repeated references occur in discourse whenever a particular person or object is mentioned or described more than once. These references are never exactly the same. The differences in the ways in which references are realised are not only due to naturally occurring variability in speech, but are also influenced by the mere fact that the information status of the referent changes when it gets repeated. For instance, when an object is mentioned a second time, it already belongs to the discourse model of speaker and addressee, and can be assumed to be common ground. Research has found that repeated references in speech are often reduced, as compared to initial references. This reduction can be acoustic (Aylett & Turk, 2004; Bard, et al., 2000; Fowler, 1988; Lindblom, 1990), lexical (Brennan & Clark, 1996; Clark & Wilkes-Gibbs, 1986), or both (Galati & Brennan, 2010). For example, a person might first be described as ‘that tall guy with the funny hat’, followed by a lexically reduced reference to ‘the hat guy’. This reduction process in repeated references extends to gesture production. It has been found that repeated references contain fewer gestures (de Ruiter, Bangerter, & Dings, 2012; Hoetjes, Koolen, Goudbeek, Krahmer, & Swerts, 2011, under revision; Jacobs & Garnham, 2007) and that gestures in repeated references are smaller (Vajrabhaya & Pederson, 2013) and less precise (Galati & Brennan, 2014; Gerwing & Bavelas, 2004; Hoetjes, et al., 2011, under revision) than gestures in initial references. We can relate these reduced repeated references to the idea that during conversation, speakers and addressees try to minimize their collaborative effort (Clark & Wilkes-Gibbs, 1986), while being as communicatively efficient as possible (Jaeger & Tily, 2011; Zipf, 1936). This is also in line with Grice’s (1975) Maxim of Quantity which states that speakers should make their contribution to the discourse “as informative as required” but “not more informative than is required”. However, speakers may also repeat words in a markedly different context, for example in a situation in which a problem occurred during interaction and it is unsure whether the reference to an object is shared by the speaker and the addressee. The question is whether in cases like this, an opposite process could also take place, in which speakers do not reduce, or might even increase, their (speech and) gesture production of repeated references, especially if doing so is communicatively necessary or efficient. In this paper we investigate a likely scenario in which a repeated reference may be expanded rather than reduced, namely when a speaker receives (negative) feedback. Effect of feedback on gesture production It can be argued that in cases of successful repeated references the addressee (implicitly) informs the speaker that less (gestural) information suffices. However, what about situations of miscommunication, or situations in which the addressee needs more information? Lindblom (1990) claims, in his theory of hyper & hypo-speech, that speakers reduce when this is possible, but hyper-articulate when this is necessary. In fact, Oviatt, Bernard and Levow (1998) found that when users of a simulated automatic speech recognition system corrected recognition errors, they tended to hyper-articulate, for example by lengthening speech segments. Providing feedback is the simplest way in which the addressee can explicitly tell the speaker that more (or less) information is required. Previous research on the effect of feedback in spoken conversation found that more details