uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems
Yuma Tsuta, Naoki Yoshinaga, Masashi Toyoda · 2020
Because open-domain dialogues allow diverse responses, basic reference-based metrics such as BLEU do not work well unless we prepare a massive reference set of high-quality responses for input utterances.To reduce this burden, a human-aided, uncertainty-aware metric, ∆BLEU, has been proposed; it embeds human judgment on the quality of reference outputs into the computation of multiplereference BLEU.In this study, we instead propose a fully automatic, uncertainty-aware evaluation method for open-domain dialogue systems, υBLEU.This method first collects diverse reference responses from massive dialogue data and then annotates their quality judgments by using a neural network trained on automatically collected training data.Experimental results on massive Twitter data confirmed that υBLEU is comparable to ∆BLEU in terms of its correlation with human judgment and that the state of the art automatic evaluation method, RUBER, is improved by integrating υBLEU.