chrF-S: Semantics Is All You Need
Ananya Mukherjee, Manish Shrivastava · 2024
Machine translation (MT) evaluation metrics like BLEU and chrF++ are widely used reference-based metrics that do not require training and are language-independent.However, these metrics primarily focus on ngram matching and often overlook semantic depth and contextual understanding.To address this gap, we introduce chrF-S (Semantic chrF++), an enhanced metric that integrates sentence embeddings to evaluate translation quality more comprehensively.By combining traditional character and word n-gram analysis with semantic information derived from embeddings, chrF-S captures both syntactic accuracy and sentence-level semantics.This paper presents our contributions to the WMT24 shared metrics task, showcasing our participation and the development of chrF-S.We also demonstrate that, according to preliminary results on the leaderboard, our metric performs on par with other supervised and LLM-based metrics.By merging semantic insights with n-gram precision, chrF-S offers a significant enhancement in the assessment of machine-generated translations, advancing the field of MT evaluation.