When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
Tao Leí · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
Large language models have become increasingly difficult to train because of the growing computation time and cost.In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling.SRU++ exhibits strong modeling capacity and training efficiency.On standard language modeling tasks such as ENWIK8, WIKI-103 and BIL-LION WORD datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to topperforming Transformer models.For instance, our model achieves a state-of-the-art result on the ENWIK8 dataset using 1.6 days of training on an 8-GPU machine.We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance.Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.1