Corpus Selection and Optimization for Statistical Machine Translation System Based on Information Retrieval Method
Qun Liu · Zhongwen xinxi xuebao · 2008
Parallel corpora are an indispensable resource for translation model training in statistical machine translation(SMT) system.Instead of collecting more and more parallel training corpora,this paper aims to improve the performance of SMT system by exploiting full potential of the existing parallel corpora.We propose an approach to select and optimize training corpus by using information retrieval method.First,sentences similar to the test text are selected to form a small and adapted training data.This allows us to get a comparable or even better performance with only a subset of the total data and the less hardware need.Second,we add the selected subset to the entire corpus to optimize the data distribution and get a better result.The experiments show that this method can effectively improve the performance of SMT system.