Quality in machine translation and human post-editing : error annotation and specifications
Lucia Comparin · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · 2017
Machine translation (MT) has been an important field of research in the last decades and is currently playing a key role in the translation market. The variable quality of results depending on various factors makes it necessary to combine MT with post-editing, to obtain high-quality translation. Post-editing is, nonetheless, a costly and time-consuming task. In order to improve the overall performance of a translation workflow involving MT, it is crucial to evaluate the quality of results produced to identify the main errors and outline strategies to address them. In this study, we assessed the results of MT and after the first human post-edition at Unbabel, a Portuguese startup that provides translation services combining MT with post-editing performed online by a community of editors. A corpus of texts translated at Unbabel from English into Italian was annotated after MT and after the first post-edition step. The data collected allowed us to identify three types of errors that are frequent and critical in terms of quality, namely “word order”, “agreement”, and “tense/mood/aspect”. Hence, correcting the errors belonging to these categories would have a major impact on the quality of translation and turn the post-editing process more accurate and efficient. The errors annotated in the corpus were analyzed in order to identify common patterns of errors, and possible solutions to address the issues identified were outlined. The MT system used at Unbabel and the tools available determined the choice to integrate information retrieved by error analysis in the Smartcheck, the tool used at Unbabel to automatically detect errors in the target text produced by the MT system and provide relevant messages to the editors. Therefore, our study focused on the definition and integration of rules in the Smartcheck to detect the most frequent and critical errors in the texts, in order to provide informative and accurate messages to the editor to aid him/her in the post-editing process.