Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act
Bao Chen, Yuanjie Wang, Zeming Liu, Yuhang Guo · 2023
Evaluation of dialogue systems requires assessing various aspects, among which appropriateness holds significance as a core element of communicative language competence.However, current evaluations heavily rely on human judgments, which are time-consuming, laborintensive, prone to biases, and lacking objectivity.In this paper, we introduce Dialogue Act Appropriateness (DAA), a novel method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses.We learn transition patterns from human-human dialogue corpora, evaluating chatbot appropriateness by measuring the similarity of their transition patterns to those observed in human-human dialogues.To validate DAA, we annotate a test dataset by manually evaluating the appropriateness of dialogues from multiple chatbot systems.The experimental results demonstrate a strong correlation between our evaluation metric and human ratings, establishing the reliability of DAA as a measure of dialogue appropriateness. 1