Dependency Analysis of Scrambled References for Better Evaluation of Japanese Translation
Hideki Isozaki, Natsume Kouchi · 2015
was proposed.RIBES measures similarity of the word order of a machine-translated sentence and that of a corresponding human-translated reference sentence.RIBES has much stronger correlation than BLEU but most Japanese sentences have alternative word orders (scrambling), and one reference sentence is not sufficient for fair evaluation.Isozaki et al. (2014) proposed a solution to this problem.This solution generates semantically equivalent word orders of reference sentences.Automatically generated word orders are sometimes incomprehensible or misleading, and they introduced a heuristic rule that filters out such bad sentences.However, their rule is too conservative and generated alternative word orders for only 30% of reference sentences.In this paper, we present a rule-free method that uses a dependency parser to check scrambled sentences and generated alternatives for 80% of sentences.The experimental results show that our method improves sentence-level correlation with human judgments.In addition, strong system-level correlation of single reference RIBES is not damaged very much.We expect this method can be applied to other languages such as German, Korean, *