Modeling Easiness for Training Transformers with Curriculum Learning
Leonardo Ranaldi, Giulia Pucci, Fabio Massimo Zanzotto · 2023
Directly learning from complex examples is generally problematic for humans and machines.Indeed, a better strategy is exposing learners to examples in a reasonable, pedagogically-motivated order.Curriculum Learning (CL) has been proposed to import this strategy when training machine learning models.In this paper, building on Curriculum Learning, we propose a novel, linguistically motivated measure to determine example complexity for organizing examples during learning.Our complexity measure -LRC-is based on length, rarity, and comprehensibility.Our resulting learning model is CL-LRC, that is, CL with LRC.Experiments on downstream tasks show that CL-LRC outperforms existing CL and non-CL methods for training BERT and RoBERTa from scratch.Furthermore, we analyzed different measures, including perplexity, loss, and learning curve of different models pre-trained from scratch, showing that CL-LRC performs better than the state-of-the-art.