An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation
Colin Cherry · 2016
The neural network joint model of translation or NNJM (Devlin et al., 2014) combines source and target context to produce a powerful translation feature.However, its softmax layer necessitates a sum over the entire output vocabulary, which results in very slow maximum likelihood (MLE) training.This has led some groups to train using Noise Contrastive Estimation (NCE), which side-steps this sum.We carry out the first direct comparison of MLE and NCE training objectives for the NNJM, showing that NCE is significantly outperformed by MLE on large-scale Arabic-English and Chinese-English translation tasks.We also show that this drop can be avoided by using a recently proposed translation noise distribution.