FedAdapter: Efficient Federated Learning for Mobile NLP
Dongqi Cai, Yaozong Wu, Shangguang Wang, Mengwei Xu · 2023
Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. To automate adapter configuration, we propose FedAdapter 1, a framework that enhances the existing FedNLP with progressive training and sideline trial. Extensive experiments show that FedAdapter can reduce FedNLP’s model convergence delay to no more than several hours.