Enhancing Pre-Training Effectiveness Through LoRA in Embedding Layers
Haeun Chun, Hyunook Yu, Mincheol Shin, Hyonjun Kang, Mucheol Kim · 2024
Text classification is the process of categorizing text into predefined labels and is a core task in natural language processing. This process enhances system efficiency in various applications such as spam email filtering, news categorization, and sentiment analysis of comments. However, fine-tuning involves training all parameters, leading to increased spatial and temporal complexity. To address this issue, Low-Rank Adaptation (LoRA) has been proposed, which reduces complexity by significantly decreasing the rank of the training parameters. In this study, we apply LoRA to the embedding layer of bidirectional encoder representations from transformers (BERT), reducing the rank of the parameters and lowering complexity. This approach effectively operates in the embedding layer and can improve performance. Experimental results show that by adding and training a new token to appear at the end of sentences, we achieved excellent performance in text classification tasks. These results demonstrate that LoRA can effectively operate in the embedding layer, suggesting its potential contribution to enhancing the performance of Transformer models.