Knowledge Distillation Enables Federated Learning: A Data-free Federated Aggregation Scheme
Jiayi Huang, Yuanyuan Zhang, Renwan Bi, Jiayin Lin, Jinbo Xiong · 2024
Applying knowledge distillation (KD) in federated learning (FL) can transfer model knowledge between clients’ local models and global model, which helps to improve the generalization of the global model. However, this requires both the clients and the server to have public data sets, which may lead to potential privacy disclosure issues. In this paper, we propose a federated data-free knowledge distillation framework, namely FedDFKD, which does not rely on any public data sets. There is a lightweight delivery model we design to learn and transfer model knowledge in different clients. During local training, the local model is jointly trained with delivery model using local data sets, and the local model feeds back its knowledge to the delivery model after it has finished its training phase in this communication round. Afterwards, the server performs global model aggregation and knowledge distillation of the delivery model. Finally, the server returns global model and distillation result to clients. We compare FedDFKD with the most representative aggregation algorithms in FL, and the results show that our method is feasible and outperforms the compared methods by between 0.1 and 3.96 percent of the global model on the MNIST dataset.