AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks
Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, Hung-yi Lee · Findings of the Association for Computational Linguistics: NAACL 2022 · 2022
Transformer-based pre-trained models with millions of parameters require large storage.Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters.In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer.Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias.The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pretrained models.We further find that Adapter-Bias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration. 1