On Losses for Modern Language Models
Stephane T. Aroca-Ouellette, Frank Rudzicz · 2020
BERT set many state-of-the-art results over varied NLU benchmarks by pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP), the latter of which has been highly criticized.In this paper, we 1) clarify NSP's effect on BERT pre-training, 2) explore fourteen possible auxiliary pre-training tasks, of which seven are novel to modern language models, and 3) investigate different ways to include multiple tasks into pre-training.We show that NSP is detrimental to training due to its context splitting and shallow semantic signal.We also identify six auxiliary pre-training tasks -sentence ordering, adjacent sentence prediction, TF prediction, TF-IDF prediction, a Fast-Sent variant, and a Quick Thoughts variant -that outperform a pure MLM baseline.Finally, we demonstrate that using multiple tasks in a multi-task pre-training framework provides better results than using any single auxiliary task.Using these methods, we outperform BERT Base on the GLUE benchmark using fewer than a quarter of the training tokens.