JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation
Xiaobo Liang, Lijun Wu, Juntao Li, Min Zhang · 2022
Transformer-based autoregressive and nonautoregressive models have played an essential role in sequence generation tasks.The autoregressive model can obtain excellent performance, while the non-autoregressive model brings fast decoding speed for inference.In this paper, we propose JANUS, a Joint Autoregressive and Non-autoregressive training method using aUxiliary losS to enhance the model performance in both AR and NAR manner simultaneously and effectively alleviate the problem of distribution discrepancy.Further, we pre-train BART with JANUS on a large corpus with minimal cost (16 GPU days) and make the BART-JANUS capable of nonautoregressive generation, demonstrating that our approach can transfer the AR knowledge to NAR.Empirically, we show our approach and BART-JANUS can achieve significant improvement on multiple generation tasks, including machine translation and GLGE benchmarks.Our code is available at Github 1 .