On the Copying Behaviors of Pre-Training for Neural Machine Translation
Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia Sam Chao, Shuming Shi, Zhaopeng Tu · 2021
Previous studies have shown that initializing neural machine translation (NMT) models with the pre-trained language models (LM) can speed up the model training and boost the model performance.In this work, we identify a critical side-effect of pre-training for NMT, which is due to the discrepancy between the training objectives of LM-based pre-training and NMT.Since the LM objective learns to reconstruct a few source tokens and copy most of them, the pre-training initialization would affect the copying behaviors of NMT models.We provide a quantitative analysis of copying behaviors by introducing a metric called copying ratio, which empirically shows that pre-training based NMT models have a larger copying ratio than the standard one.In response to this problem, we propose a simple and effective method named copying penalty to control the copying behaviors in decoding.Extensive experiments on both indomain and out-of-domain benchmarks show that the copying penalty method consistently improves translation performance by controlling copying behaviors for pre-training based NMT models.