Cometoid: Distilling Strong Reference-based Machine Translation Metrics into Even Stronger Quality Estimation Metrics
Thamme Gowda, Tom Kocmi, Marcin Junczys-Dowmunt · 2023
This paper describes our submissions to the 2023 Conference on Machine Translation (WMT23) Metrics shared task.Knowledge distillation is commonly used to create smaller student models that mimic a larger teacher model while reducing the model size and hence inference cost in production.In this work, we apply knowledge distillation to machine translation evaluation metrics and distill existing reference-based teacher metrics into reference-free (quality estimation; QE) student metrics.We mainly focus on students of Unbabel's COMET22 reference-based metric.When evaluating on the official WMT22 Metrics evaluation task, our distilled Cometoid QE metrics outperform all other QE metrics on that set while matching or out-performing the reference-based teacher metric.Our metrics never see the human ground-truth scores directly -only the teacher metric was trained on human scores by its original creators.We also distill ChrF sentence-level scores into a neural QE metric and find that our reference-free (and fully human-score-free) student metric ChrFoid outperforms its teacher metric by over 7% pairwise accuracy on the same WMT22 task, rivaling other existing QE metrics. 1