Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT
Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser · 2020
Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural machine translation (UNMT) system yields stateof-the-art results.When limited data is available for one language, however, this method leads to poor translations.We present an effective approach that reuses an LM that is pretrained only on a high-resource language.The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model.To reuse the pretrained LM, we have to modify its predefined vocabulary, to account for the new language.We therefore propose a novel vocabulary extension method.Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions.