Learning Better Masking for Better Language Model Pre-training
Dongjie Yang, Zhuosheng Zhang, Hai Yan Zhao · 2023
Masked Language Modeling (MLM) has been widely used as the denoising objective in pretraining language models (PrLMs).Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training.However, the model may receive a complicated impact from pre-training status, which changes accordingly as training time goes on.In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings.We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks.Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pretraining 1 .