Weakly Supervised Turn-level Engagingness Evaluator for Dialogues
Shaojie Jiang, Svitlana Vakulenko, Maarten de Rijke · 2023
Engagingness is an important measurement for evaluating open-domain conversational systems. The standard approach to evaluating dialogue engagingness is by measuring conversation turns per session (CTPS), which implies that the dialogue length is the main predictor of the user engagement with a dialogue system. The main limitation of CTPS is that it can only be measured at the session level, i.e., once the dialogue is over. But a dialogue system has to continuously monitor user engagement throughout the dialogue session as well. Existing approaches to measuring turn-level engagingness require human annotations for training. We pioneer an alternative approach, Weakly Supervised Engagingness Evaluator (WeSEE), which uses the remaining depth for each turn as a heuristic weak label for engagingness. Weakly Supervised Engagingness Evaluator (WeSEE) does not require human annotations and also relates closely to CTPS, thus serving as a good learning proxy for this metric. We show that WeSEE achieves the new state-of-the-art results on the Fine-grained Evaluation of Dialog dataset (0.38 Spearman correlation coefficient) and the DailyDialog dataset (0.62 Spearman correlation coefficient).