End-to-end evaluation of a speech-to-speech translation system in TC-STAR
Olivier Hamon, Djamel Mostefa, Khalid Choukri · 2007
The paper describes an evaluation methodology to evaluate speech-to-speech translation systems and their results. The evaluation scheme uses questionnaires filled in by human judges for addressing the adequacy and fluency of audio translation outputs and was applied in the second TC-STAR evaluation campaign. The same evaluation methodology is carried out both on the outputs of an automatic system and on human translations produced by professional interpreters. The obtained results show that the professional interpreters obtain better results on fluency. But surprisingly, the automatic systems results on adequacy are higher than for the human translator. But this has to be reconsidered by the fact that humans have to translate in a real time, and select important information, while an automatic system tends to translate all the information.