Towards document-level human MT evaluation: On the Issues of annotator agreement, effort and misevaluation
Sheila Castilho · Dublin City University Open Access Institutional Repository (Dublin City University) · 2021
Document-level human evaluation of machine translation (MT) has been raising interest in the community. However, little is known about the issues of using document-level methodologies to assess MT quality. In this article, we compare the inter-annotator agreement (IAA) scores, the effort to assess the quality in different document-level methodologies, and the issue of misevaluation when sentences are evaluated out of context.