On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation
Sheila Castilho · 2020
Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of "human parity" (Toral et al., 2018;Läubli et al., 2018) with document-level human evaluations have been published.Yet, little is known about best practices regarding human evaluation of machine translation at the documentlevel.This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups.We report results of the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking, along with the effort needed to perform the different tasks.To best of our knowledge, this is the first study of its kind.