Regularizing Forward and Backward Decoding to Improve Neural Machine Translation
Zhen Chao Yang, Laifu Chen, Le-Minh Nguyen · 2018
The common recurrent neural network (RNN) based sequential decoder of the neural machine translation (NMT) model can only translate from one direction, which makes the model overfit in the forward direction and leaves backward information of target sentences unexploited. We propose to use a regularization loss to encourage NMT decoder to exploit bidirectional information of target sentences. Beside of forward decoding, we train an extra set of decoding components to predict translation from backward, and use regularization to enforce the forward and backward hidden states at the same time step have connection. During training phase, the forward hidden states can encode future information from the backward hidden states; while during test phase, we only use the enhanced forward decoding components to translate. Our empirical experiments demonstrated that our approach can significantly improve the performance on WMT German-English and English-Chinese translation tasks in terms of NIST and BLEU score.