How to Train BERT with an Academic Budget
Peter Izsak, Moshe Berchansky, Omer Levy · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
While large language models à la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few wellfunded industry labs can afford.How can one train such models with a more modest budget?We present a recipe for pretraining a masked language model in 24 hours using a single lowend deep learning server.We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT BASE on GLUE tasks at a fraction of the original pretraining cost. 1