Improving Autoregressive NMT with Non-Autoregressive Model
Long Zhou, Jiajun Zhang, Chengqing Zong · 2020
Autoregressive neural machine translation (NMT) models are often used to teach nonautoregressive models via knowledge distillation.However, there are few studies on improving the quality of autoregressive translation (AT) using non-autoregressive translation (NAT).In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model.Specifically, under the semantic guidance of source-side context captured by the encoder, the nonautoregressive decoder (NAD) first learns to generate target-side hidden state sequence in parallel.Then the autoregressive decoder (AD) performs translation from left to right, conditioned on source-side and target-side hidden states.Since AD has global information generated by low-latency NAD, it is more likely to produce a better translation with less time delay.Experiments on WMT14 En⇒De, WMT16 En⇒Ro, and IWSLT14 De⇒En translation tasks demonstrate that our framework achieves significant improvements with only 8% speed degeneration over the autoregressive NMT.