Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
Mercedes-Benz AG, Sindelfingen, Germany, Liu Ye, Wolfgang Maier, Mercedes-Benz AG, Sindelfingen, Germany, Wolfgang Minker, Stefan Ultes, Mercedes-Benz AG, Sindelfingen, Germany · 2021
This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems.While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language.By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT.In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.