MSLC24: Further Challenges for Metrics on a Wide Landscape of Translation Quality
Rebecca Knowles, Samuel Larkin, Chi-kiu Lo · 2024
In this second edition of the Metric Score Landscape Challenge (MSLC), we examine how automatic metrics for machine translation perform on a wide variety of machine translation output, ranging from very low quality systems to the types of high-quality systems submitted to the General MT shared task at WMT.We also explore metric results on specific types of data, such as empty strings, wrong-or mixedlanguage text, and more.We raise several alarms about inconsistencies in metric scores, some of which can be resolved by increasingly explicit instructions for metric use, while others highlight technical flaws.