eBLEU: Unexpectedly Good Machine Translation Evaluation Using Simple Word Embeddings
Muhammad ElNokrashy, Tom Kocmi · 2023
We propose eBLEU, a metric inspired by BLEU metric that uses embedding similarities instead of string matches.We introduce meaning diffusion vectors to enable matching n-grams of semantically similar words in a BLEU-like algorithm, using efficient, non-contextual word embeddings like fastText.On WMT23 data, eBLEU beats BLEU and ChrF by around 3.8% system-level score, approaching BERTScore at -0.9% absolute difference.In WMT22 scenarios, eBLEU outperforms f101spBLEU and ChrF in MQM by 2.2%-3.6%.Curiously, on MTurk evaluations, eBLEU surpasses past methods by 3.9%-8.2%(f200spBLEU, COMET-22).eBLEU presents an interesting middle-ground between traditional metrics and pretrained metrics.