Perception vs. reality: measuring machine translation post-editing productivity
Federico Gaspari, Antonio Toral, Sudip Kumar Naskar, Declan Groves, Andy Way · 2014
This paper presents a study of user-perceived vs real machine translation (MT) post-editing effort and productivity gains, focusing on two bidirectional language pairs: English— German and English—Dutch. Twenty experienced media professionals post-edited statisti-cal MT output and also manually translated comparative texts within a production environ-ment. The paper compares the actual post-editing time against the users ’ perception of the effort and time required to post-edit the MT output to achieve publishable quality, thus measuring real (vs perceived) productivity gains. Although for all the language pairs users perceived MT post-editing to be slower, in fact it proved to be a faster option than manual translation for two translation directions out of four, i.e. for Dutch→English, and (marginal-ly) for English→German. For further objective scrutiny, the paper also checks the correla-tion of three state-of-the-art automatic MT evaluation metrics (BLEU, METEOR and TER) with the actual post-editing time. 1