RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
Albert Zeyer, Tamer Alkhouli, Hermann Ney · 2018
We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure Tensor-Flow beam search decoder.We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks.Promising preliminary results on max.expected BLEU training are presented.We obtain state-of-the-art models trained on the WMT 2017 German↔English translation task.We also present end-to-end model results for speech recognition on the Switchboard task.The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.