In-Context Evaluation of Unsupervised Dialogue Act Models for Tutorial Dialogue
Aysu Ezen-Can, Kristy Elizabeth Boyer · 2013
Unsupervised dialogue act modeling holds great promise for decreasing the development time to build dialogue systems. Work to date has utilized manual annotation or a synthetic task to evaluate unsupervised dialogue act models, but each of these evaluation approaches has substantial limitations. This paper presents an incontext evaluation framework for an unsupervised dialogue act model within tutorial dialogue. The clusters generated by the model are mapped to tutor responses by a handcrafted policy, which is applied to unseen test data and evaluated by human judges. The results suggest that incontext evaluation may better reflect the performance of a model than comparing against manual dialogue act labels. 1