Federated Learning Method Based on Knowledge Distillation and Deep Gradient Compression
Haiyan Cui, Junping Du, Jiang Yang, Yue Wang, Runyu Yu · 2021
Federated learning is a new type of multi-agency collaborative training model paradigm, which is widely used in many fields, among which communication overhead is a key issue. In order to reduce the amount of data transmitted in the communication process, we propose a federated learning algorithm based on knowledge distillation and deep gradient compression (Fed-KDDGC-SGD). First, we use local data on the client to train the teacher network, and then use the soft labels generated by the teacher network to train the student network and upload the gradient to the central server during the training process. In order to further reduce the communication bandwidth occupied by sending the gradient, the deep gradient compression algorithm is used to compress the gradient vector, and only the gradient value of the top R% of the absolute value is sent. The experimental results show that the improved federated learning algorithm effectively reduces the communication overhead and has certain practical significance.