BLEU Is Not the Colour : How Optimising BLEU Reduces Translation Quality
Aaron Smith, Christian Hardmeier, Jörg Tiedemann · 2014
In this paper we present experiments in statistical machine translation (SMT) where the BLEU score is optimised with the aim of providing high-quality translations to use as data points in parameter tuning. This is achieved by running the document-level SMT decoder Docent in BLEU-decoding mode, where proposed changes to the translation of a document are checked against a reference, and only accepted if they increase BLEU. The expected increase in translation quality is however not forthcoming. Our results suggest that BLEU is not always a reliable metric; conversely, directly optimising for BLEU can seriously damage translation quality.