FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models
Xinge Ma, Jiangming Liu, Jin Wang, Xuejie Zhang · 2023
The growing concerns and regulations surrounding the protection of user data privacy have necessitated decentralized training paradigms.To this end, federated learning (FL) is widely studied in user-related natural language processing (NLP).However, it suffers from several critical limitations including extensive communication overhead, inability to handle heterogeneity, and vulnerability to white-box inference attacks.Federated distillation (FD) is proposed to alleviate these limitations, but its performance is faded by confirmation bias.To tackle this issue, we propose Federated Interactive Distillation (FedID), which utilizes a small amount of labeled data retained by the server to further rectify the local models during knowledge transfer.Additionally, based on the GLUE benchmark, we develop a benchmarking framework across multiple tasks with diverse data distributions to contribute to the research of FD in NLP community.Experiments show that our proposed Fe-dID framework achieves the best results in homogeneous and heterogeneous federated scenarios.The code for this paper is available at: https://github.com/maxinge8698/FedID.