LAYERED: Metric for Machine Translation Evaluation
Shubham Gautam, Pushpak Bhattacharyya · 2014
This paper describes the LAYERED metric which is used for the shared WMT'14 metrics task.Various metrics exist for MT evaluation: BLEU (Papineni, 2002), ME-TEOR (Alon Lavie, 2007), TER (Snover, 2006) etc., but are found inadequate in quite a few language settings like, for example, in case of free word order languages.In this paper, we propose an MT evaluation scheme that is based on the NLP layers: lexical, syntactic and semantic.We contend that higher layer metrics are after all needed.Results are presented on the corpora of ACL-WMT, 2013 and 2014.We end with a metric which is composed of weighted metrics at individual layers, which correlates very well with human judgment.