LEBERT :Lite and Efficiently Optimized BERT PRetraining Approach
Priyanka Yadav, Anjali R. Sharma · International Journal of Advanced Research in Science Communication and Technology · 2024
The extensive generalization of these models can lead to overfitting, causing the model to perform poorly on unseen data and thereby not realizing its full potential. To address this challenge systematically, we propose a novel approach for lightweight and efficient fine-tuning of BERT (Bidirectional Encoder Representations from Transformers) that aims to achieve improved generalization and harness the maximum capabilities of BERT. Our proposed approach incorporates various regularization techniques designed to adaptively manage the model's complexity. We plan to conduct experiments using this approach across various NLP tasks, including GLUE (Wang et al., 2019), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).