A Diagnostic Evaluation Approach Targeting MT Systems for Indian Languages
Renu Balyan, Sudip Kumar Naskar, Antonio Toral, Niladri Chatterjee · 2012
This paper addresses diagnostic evaluation of machine translation (MT) systems for Indian languages, English to Hindi translation in particular. Evaluation of MT output is an important but difficult task. The difficulty arises primarily from some inherent characteristics of the language pairs, which range from simple word-level discrepancies to more difficult structural variations for Hindi from English, such as reduplication of words, free word order etc. The proposed scheme is based on identification of linguistic units (often referred to as checkpoints). We use the diagnostic evaluation tool DELiC4MT to analyze the contribution of various PoS classes for different categories. We further suggest some additional checkpoints based on named entities, ambiguous words, word order and inflections that are relevant for the evaluation of Hindi. The evaluation of these checkpoints provides a detailed analysis and helps in monitoring how an MT system handles these linguistic phenomena as well. This also provides valuable feedback to MT developers as to where the system is performing poorly and how the output can possibly be improved. The effectiveness of the approach was tested on 5 English to Hindi MT systems and it was observed that the system-level DELiC4MT scores correlate well with the scores produced by the most commonly used automatic evaluation metrics (BLEU, NIST, METEOR and TER) while providing finer-grained information.