Parameter-Efficient Fine-Tuning without Introducing New Latency
Baohao Liao, Yan Meng, Christof Monz · 2023
Parameter-efficient fine-tuning (PEFT) of pretrained language models has recently demonstrated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable parameters, and consequently addressing the storage and communication constraints.Nonetheless, various PEFT methods are limited by their inherent characteristics.In the case of sparse fine-tuning, which involves modifying only a small subset of the existing parameters, the selection of fine-tuned parameters is task-and domain-specific, making it unsuitable for federated learning.On the other hand, PEFT methods with adding new parameters typically introduce additional inference latency.In this paper, we demonstrate the feasibility of generating a sparse mask in a task-agnostic manner, wherein all downstream tasks share a common mask.Our approach, which relies solely on the magnitude information of pre-trained parameters, surpasses existing methodologies by a significant margin when evaluated on the GLUE benchmark.Additionally, we introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation, thereby achieving identical inference speed to that of full finetuning.Through extensive experiments, our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.1