Communication-Efficient Generalized Neuron Matching for Federated Learning
Sixu Hu, Qinbin Li, Bingsheng He · 2023
Federated Learning (FL) is a popular distributed machine learning paradigm that allows multiple participants to collaboratively train a model without sharing the raw data. In each round of FL, all participants train local models in parallel, and a server aggregates the local models to create the global model. Neuron matching is a promising aggregation method that utilizes permutational invariance to improve the quality of the global model and reduces communication costs. However, existing neural matching approaches have two strict limitations: 1) They are only applicable to models with simple sequential networks and cannot support advanced models such as ResNet. 2) Their approaches significantly increase the model size after matching, which causes massive memory and communication costs as the training proceeds. In this paper, we propose a novel neuron matching algorithm called federated generalized matched averaging (FedGMA) to mitigate these limitations. Our experiments show that the proposed method is applicable to complex network structures such as ResNet and InceptionNet, and achieves accuracy comparable to algorithms that directly average the entire model. It also reduces communication costs and demonstrates robustness in data heterogeneity scenarios.