Pretrained Language Models and Backtranslation for English-Basque Biomedical Neural Machine Translation
Iñigo Jauregi Unanue, Massimo Piccardi · 2020
This paper describes the machine translation systems proposed by the University of Technology Sydney Natural Language Processing (UTS NLP) team for the WMT20 English-Basque biomedical translation tasks.Due to the limited parallel corpora available, we have opted to train a BERT-fused NMT model that leverages the use of pretrained language models.Furthermore, we have augmented the training corpus by backtranslating monolingual data.Our experiments show that NMT models in low-resource scenarios can benefit from combining these two training techniques, with improvements of up to 6.16 BLEU percentage points in the case of biomedical abstract translations.