Topic Shift in Efficient Discourse Production
Ting Qian, T. Florian Jaeger · eScholarship (California Digital Library) · 2011
Speakers have been hypothesized to organize discourse content so as to achieve communicative efficiency.Previous work has focused on indirect tests of the hypothesis that speakers aim to keep per-word entropy constant across discourses to achieve communicative efficiency (Genzel & Charniak, 2002).We present novel and more direct evidence by examining the role of topic shift in discourse planning.If speakers aim for constant per-word entropy, they should encode less unconditional per-word entropy (as estimated based on only sentenceinternal cues) following topic shifts, as there is less relevant context to condition on.Applying latent topic modeling to a large set of English texts, we find that speakers are indeed sensitive to the recent topic structure in the predicted way.