Appraise: An Open-Source Toolkit for Manual Evaluation of Machine Translation Output
Christian Federmann · 2012
Abstract. We describe Appraise, an open-source toolkit which can be used to do manual evaluation of Machine Translation output. Appraise allows to collect human judgments on translation output, implementing annotation tasks such as 1) translation quality checking, 2) ranking of translations, 3) error classification, and 4) manual post-editing. It uses an extensible format for import/export of data and can easily be adapted to new annotation tasks. The annotation tasks are explained in more detail in the paper. The current version of Appraise also includes automatic computation of inter-annotator agreement scores resulting in quick access to evaluation results. Appraise has successfully been used for a wide variety of research projects.