Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation
Huda Khayrallah, Brian J. Thompson, Kevin Duh, Philipp Koehn · 2018
Supervised domain adaptation-where a large generic corpus and a smaller indomain corpus are both available for training-is a challenge for neural machine translation (NMT).Standard practice is to train a generic model and use it to initialize a second model, then continue training the second model on in-domain data to produce an in-domain model.We add an auxiliary term to the training objective during continued training that minimizes the cross entropy between the indomain model's output word distribution and that of the out-of-domain model to prevent the model's output from differing too much from the original out-ofdomain model.We perform experiments on EMEA (descriptions of medicines) and TED (rehearsed presentations), initialized from a general domain (WMT) model.Our method shows improvements over standard continued training by up to 1.5 BLEU.