B-Pet: The PET Model with Parameter-Efficient Learning
Qi Zheng, Haizheng Yu · 2023
In recent years, under the trend of training models in big data, Few-shot learning (FSL) which aims to learn models to solve problems with a few samples has also achieved good results on many data sets. In fact, acquiring high-quality training samples is expensive in many aspects, but FSL can save the overhead costs. Among FSL models, the PET model combines semi-supervised learning, prompt learning and knowledge distillation based on the pre-training language model. However, in fine-turning the PET model has the disadvantages that consumes a lot of resources and time and requires heavy costs of storage for model preservation. Therefore, this paper proposes the B-pet model, which freezes most of the training parameters and only trains bias parameters during fine-turning process, significantly reducing the storage consumption of the model for downstream tasks. We used six data sets with$\vert \tau \vert=\mathbf{10},\ \mathbf{50},\ \mathbf{100}$and three different data training models respectively. The results show that four data sets on the B-pet model performed better than original PET model training. It is obvious that in the memory-constrained environment deployment, multitasking fine-tunes models have practical value. It also proved that most semi-supervised models with fixed parameters are realizable.