Automatic Metric Validation for Grammatical Error Correction
Leshem Choshen, Omri Abend · 2018
Metric validation in Grammatical ErrorCorrection (GEC) is currently done by observing the correlation between human and metric-induced rankings.However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement.We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices.Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M 2 metric fares poorly on corpus-level ranking.Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.