Dependency-based Automatic Enumeration of Semantically Equivalent Word Orders for Evaluating Japanese Translations
Hideki Isozaki, Natsume Kouchi, Tsutomu Hirao · 2014
Scrambling is acceptable reordering of verb arguments in languages such as Japanese and German. In automatic eval-uation of translation quality, BLEU is the de facto standard method, but BLEU has only very weak correlation with hu-man judgements in case of Japanese-to-English/English-to-Japanese translations. Therefore, alternative methods, IMPACT and RIBES, were proposed and they have shown much stronger correlation than BLEU. Now, RIBES is widely used in recent papers on Japanese-related transla-tions. RIBES compares word order of MT output with manually translated reference sentences but it does not regard scram-bling at all. In this paper, we present a method to enumerate scrambled sentences from dependency trees of reference sen-tences. Our experiments based on NTCIR Patent MT data show that the method im-proves sentence-level correlation between RIBES and human-judged adequacy. 1