Improving evaluation and optimization of MT systems against MEANT
Chi-kiu Lo, Philipp Dowling, Dekai Wu · 2015
We show that, consistent with MEANTtuned systems that translate into Chinese, MEANT-tuned MT systems that translate into English also outperforms BLEUtuned systems across commonly used MT evaluation metrics, even in BLEU.The result is achieved by significantly improving MEANT's sentence-level ranking correlation with human preferences through incorporating a more accurate distributional semantic model for lexical similarity and a novel backoff algorithm for evaluating MT output which automatic semantic parser fails to parse.The surprising result of MEANT-tuned systems having a higher BLEU score than BLEU-tuned systems suggests that MEANT is a more accurate objective function guiding the development of MT systems towards producing more adequate translation.