A Holistic Approach to Reference-Free Evaluation of Machine Translation
Hanming Wu, Wenjuan Han, Hui Di, Yufeng Chen, Jinan Xu · 2023
Traditional machine translation evaluation relies on references written by humans.While reference-free evaluation gets rid of the constraints of labor-intensive annotations, it can pivot easily to new domains and is more scalable.In this paper, we propose a referencefree evaluation approach that characterizes evaluation as two aspects: (1) fluency: how well the candidate translation conforms to normal human language usage; (2) faithfulness: how well the candidate translation reflects the source data.We further split the faithfulness into word-level and sentence-level.Extensive experiments spanning WMT18/19/21 Metrics segment-level daRR and MQM datasets demonstrate that our proposed reference-free approach, ReFreeEval, outperforms SOTA reference-free metrics like YiSi-2, SentSim and BERTScore-MKD in most language directions.The code can be found at ReFreeEval Repo 1 .