Accelerating Federated Learning with Two-phase Gradient Adjustment
Jiajun Wang, Yingchi Mao, Xiaoming He, Tong Zhou, Jun Wu, Jie Wu · 2023
With the advent of the Internet of Things (IoT) era and 5G, ubiquitous sensing devices (e.g., smartphones, surveillance sites, and security cameras) have been widely used in various fields, resulting in the generation of a huge amount of monitoring data. The rise of federated learning makes it possible to leverage monitoring data to train deep neural networks through cloud-edge collaboration without compromising privacy. However, the non identically and independently distributed (called Non-IID) data collected by IoT devieces creates a client drift phenomenon, resulting in a slow convergence of the global model. To this end, we propose a new Federated learning framework based on Gradient Variance Reduction with a correction weight control mechanism and Global gradient descent with Momentum, named FedGVRGM to conduct gradient correction and reduce the negative impacts of prediction parameters. Specifically, in the local training phase, FedGVRGM combines gradient variance reduction with a correction weight control mechanism to further correct the local model parameters, thus reducing the dispersion of model parameters among clients. In the global aggregation phase, FedGVRGM integrates the historical change states of the global model through the gradient descent with momentum to reduce the oscillations and improve the convergence speed of the global model. We refer to the above methods of gradient adjustment in the local and global training phases as FedGVR and FedGM, respectively. Numerous evaluations are conducted on CIFAR-100, CIFAR-10, and MNIST datasets to prove that FedGVRGM has a faster convergence rate than other stateof-the-art approaches such as Federated Averaging (FedAvg), FedProx, FedReg, FedGVR, and FedGM.