Distilled BERT Model in Natural Language Processing
Yazdan Zandiye Vakili, Avisa Fallah, Hedieh Sajedi · 2024
This paper reviews the evolution of Natural Language Processing (NLP) models, concentrating on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models deliver robust performance while being optimized for use on devices with limited computational resources, enabling the application of advanced NLP features in practical scenarios.