Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation?
Hiroto Tamura, Tosho Hirasawa, Hwichan Kim, Mamoru Komachi · 2023
Pre-training masked language models (MLMs) with artificial data has been proven beneficial for several natural language processing tasks such as natural language understanding and summarization; however, it has been less explored for neural machine translation (NMT).A previous study revealed the benefit of transfer learning for NMT in a limited setup, which differs from MLM.In this study, we prepared two kinds of artificial data and compared the translation performance of NMT when pretrained with MLM.In addition to the random sequences, we created artificial data mimicking token frequency information from the real world.Our results showed that pre-training the models with artificial data by MLM improves translation performance in low-resource situations.Additionally, we found that pre-training on artificial data created considering token frequency information facilitates improved performance.