A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English
Toshiyuki Takezawa, Fumiaki Sugaya, Akio Yokoo, Seiichi Yamamoto · 1999
ATR-MATRIX is a multi-lingual speech-to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel ar-rangement domain. In this paper, we pro-pose a new evaluation method for speech translation systems. Our current focus is on measuring the robustness of a language translation sub-system, with quick calcula-tion and low cost. Therefore, we calculate the difference between the translation out-put from transcription texts and the trans-lation output from input speech by a dy-namic programming method. We present the first trial experiment of this method applied to our Japanese-to-English speech translation system. We also provide re-lated discussions on such points as error analysis and the relationship between the proposed method and translation quality evaluation manually done by humans. 1