RED: A Reference Dependency Based MT Evaluation Metric
Hui Yu, Xiaofeng Wu, Jun Xie, Wenbin Jiang, Qun Liu, Shouxun Lin · 2014
Most of the widely-used automatic evaluation metrics consider only the local fragments of the references and translations, and they ignore the evaluation on the syntax level. Current syntax-based evaluation metrics try to introduce syntax information but suffer from the poor pars-ing results of the noisy machine translations. To alleviate this problem, we propose a novel dependency-based evaluation metric which only employs the dependency information of the ref-erences. We use two kinds of reference dependency structures: headword chain to capture the long distance dependency information, and fixed and floating structures to capture the local con-tinuous ngram. Experiment results show that our metric achieves higher correlations with human judgments than BLEU, TER and HWCM on WMT 2012 and WMT 2013. By introducing extra linguistic resources and tuning parameters, the new metric gets the state-of-the-art performance which is better than METEOR and SEMPOS on system level, and is comparable with METEOR on sentence level on WMT 2012 and WMT 2013. 1