A Detailed Comparative Analysis of Automatic Neural Metrics for Machine Translation: BLEURT & BERTScore
Aniruddha Mukherjee, Vikas Hassija, Vinay Chamola, Karunesh Kumar Gupta · IEEE Open Journal of the Computer Society · 2025
Bleurta recently introduced metric that employsBert, a potent pre-trained language model to assess how well candidate translations compare to a reference translation in the context of machine translation outputs. While traditional metrics likeBleurely on lexical similarities,BleurtleveragesBert's semantic and syntactic capabilities to provide more robust evaluation through complex text representations. However, studies have shown thatBert, despite its impressive performance in natural language processing tasks can sometimes deviate from human judgment, particularly in specific syntactic and semantic scenarios. Through systematic experimental analysis at the word level, including categorization of errors such as lexical mismatches, untranslated terms, and structural inconsistencies, we investigate howBleurthandles various translation challenges. Our study addresses three central questions: What are the strengths and weaknesses ofBleurt, how do they align withBert's known limitations, and how does it compare with the similar automatic neural metric for machine translation,BERTScore? Using manually annotated datasets that emphasize different error types and linguistic phenomena, we find thatBleurtexcels at identifying nuanced differences between sentences with high overlap, an area whereBERTScoreshows limitations. Our systematic experiments, provide insights for their effective application in machine translation evaluation.