Towards an automated evaluation of an embedded MT system.
Jamal Laoudi, Calanda R. Tate, Clare R. Voss · 2004
Abstract. How can recent advances in automating the evaluation of machine translation (MT) engines be applied to automate the evaluation of more complex embedded MT systems? In this paper, we describe initial evaluation testing of FALCon, an embedded MT system where hard-copy documents are scanned into bit-mapped images, converted into online text files via optical character recognition (OCR) software, and then translated by an MT engine from one natural language into another. Our challenge with the ongoing support of FALCon systems is to evaluate when the replacement of a particular component with a new or upgraded product will yield significant improvements over the baseline system performance. In this paper we address the following questions: (i) how can we automate the baseline end-to-end evaluation of this system? and (ii) what is the relation between the accuracy of the individual components and their end-to-end accuracy, that could be used to model the system performance? 1.