A Statistical Analysis of Automated MT Evaluation Metrics for Assessments in Task-Based MT Evaluation
Calandra R. Tate · 2008
This paper applies nonparametric statistical techniques to Machine Translation (MT) Eval-uation using data from a large scale task-based study. In particular, the relationship between human task performance on an information extraction task with translated documents and well-known automated translation evaluation metric scores for those documents is stud-ied. Findings from a correlation analysis of this connection are presented and contrasted with current strategies for evaluating transla-tions. An extended analysis that involves a novel idea for assessing partial rank correla-tion within the presence of grouping factors is also discussed. This work exposes the limi-tations of descriptive statistics generally used in this area, mainly correlation analysis, when using automated metrics for assessments in task handling purposes. 1