Generation of Parallel Corpus for Low Resource Language Translation
Long Hai Trieu, Ashwin Ittoo · Open Repository and Bibliography (University of Liège) · 2018
Large parallel corpora play an essential role in machine translation and other cross-lingual natural language processing tasks.Training high quality translation systems requires large parallel corpora up to million sentence pairs, which are mostly available on European languages and several other languages paired with English.For Southeast Asian languages, there are few or even no existing parallel corpus, which causes the research of machine translation on these languages more challenging.Although there are several efforts in building parallel corpora manually or collecting from available parallel resources, the corpus size is still limited.In this work, we introduce a multilingual parallel corpus containing 2.5 millions parallel sentences on ten language pairs of several Southeast Asian languages among Filipino, Malay, Indonesian, Vietnamese and these languages paired with English.The corpus is automatically built from the abundantly available Wikipedia resource.On the document level alignment, we utilize the available Wikidump inter-language link record information to extract pairs of article titles then article texts.On the sentence level alignment, we employ a powerful and independent sentence aligner to extract parallel sentences.To evaluate the corpus on machine translation, we conducted experiments using the state-of-the-art Transformer model for building translation systems and evaluated on two manually created corpora, which are the Asian Language Treebank and the IWSLT 2015 shared task.Experimental results on various settings show that the corpus can help to significantly improve the translation performance on these low-resource languages.Additionally, the framework to build the corpus is language-independent, which can be extended to build parallel corpus for other language pairs.The corpus and code are publicly available at https://github.com/trieuhl/mt-wiki.