Improving Federated Learning on Heterogeneous Data via Serial Pipeline Training and Global Knowledge Regularization
Yiyang Luo, Ting Lu, Shan Chang, Bingyue Wang · 2023
Federated learning is a distributed machine learning paradigm that resolves the conflict between training requirements and client data privacy. There are some challenges for federated learning, such as data heterogeneity and communication load, which lead to the global model bias and slow convergence. In this work, we address the problem of data heterogeneity and communication load from a novel perspective, which is named FedSPARK. 1) We propose a new federated learning interaction training strategy, serial pipeline training (SPT). SPT changes the local training of a single client to serial training of multiple clients, which improves the performance of the global model on heterogeneous data. 2) We propose the global knowledge regularization (GKR) which is inspired by continuous learning. GKR can reduce the bias of the client local model by building global knowledge. Through theoretical analysis and experiments on multiple datasets, we show that our approaches greatly reduce the computation of clients and the amount of communication between clients and server, and improve the efficiency of federated learning compared to existing methods.