Metrics for Evaluation of Word-level Machine Translation Quality Estimation
Varvara Logacheva, Michał Łukasik, Lucia Specia · 2016
The aim of this paper is to investigate suitable evaluation strategies for the task of word-level quality estimation of machine translation.We suggest various metrics to replace F 1 -score for the "BAD" class, which is currently used as main metric.We compare the metrics' performance on real system outputs and synthetically generated datasets and suggest a reliable alternative to the F 1 -BAD score -the multiplication of F 1 -scores for different classes.Other metrics have lower discriminative power and are biased by unfair labellings.