Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning
Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, Xuanjing Huang · 2023
Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks.Although prompt tuning has been shown to match the performance of full model tuning when training data is sufficient, it tends to struggle in few-shot learning settings.In this paper, we present Multi-task Pre-trained Modular Prompt (MP 2 ) to boost prompt tuning for few-shot learning.MP 2 is a set of combinable prompts pre-trained on 38 Chinese tasks.On downstream tasks, the pre-trained prompts are selectively activated and combined, leading to strong compositional generalization to unseen tasks.To bridge the gap between pre-training and fine-tuning, we formulate upstream and downstream tasks into a unified machine reading comprehension task.Extensive experiments under two learning paradigms, i.e., gradient descent and black-box tuning, show that MP 2 significantly outperforms prompt tuning, full model tuning, and prior prompt pretraining methods in few-shot settings.In addition, we demonstrate that MP 2 can achieve surprisingly fast and strong adaptation to downstream tasks by merely learning 8 parameters to combine the pre-trained modular prompts.