Context-based evaluation of MT systems: principles and tools
Maghi King, Andréi Popescu-Belis, Paula Estrella · 2007
This tutorial introduces a principled approach to the evaluation of MT systems in their context of use. A method for defining contextual evaluation plans will be described, along with an interactive tool supporting the definition of such plans. This tutorial provides the audience with knowledge and tools for MT evaluation, which will enable them to better justify their choices of particular MT systems, or to argue about the quality of MT systems in a more principled way. Framework for Machine Translation Evaluation (FEMTI) FEMTI is an achievement of the Evaluation Working Group of the ISLE European project. FEMTI relates the quality model used to evaluate an MT system to the context of use of the system. The initial version of FEMTI, which synthesized many of the previous efforts to structure the concepts and metrics used in MT evaluation, was quite complex to use. Therefore, a web-based, user-friendly tool supporting the design of a contextual quality model was implemented, and is now available online at