FedLOC:A Layer Output Based Compression Algorithm for Federated Learning
Peng Ouyang, Danyang Xiao, Jieying Zhou, Weigang Wu · 2024
Communication cost is a main challenge in Federated Learning (FL). Gradient sparsification is one of the effective ways to reduce communication data volumes by allowing clients to send only a small portion of gradient elements to the server. Existing gradient sparsification methods, such as Top-K, analyze and decide the contribution of gradient elements for uploading by comparing their values. However, gradient elements with large values may not necessarily contain more information. To improve the accuracy while effectively reducing communication costs, in this paper, we propose a novel gradient sparsification-based FL algorithm called FedLOC, which sparsifies gradients not only based on the value of the gradient elements but also on the layer output of the local model. In particular, we design a two-phase compression operation to reduce the communication data volumes. First, FedLOC sparsifies gradients based on their values to construct initial compressed gradients. Subsequently, since the performance of the local model is related to the output of each layer, FedLOC analyzes the contribution of initial compressed gradient elements and constructs masks for the second sparsification operation based on each layer’s extend output derived from the original layer output. Convergence analysis and experimental results show that FedLOC can achieve superior performance in terms of accuracy and convergence when effectively reducing data communication volumes.