Permutation mask: a combined gradient sparsification for federated learning
Shiqi Zhou, Yongdao Zhou · Journal of nonparametric statistics · 2024
Large-scale distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. This situation gets even worse with training on mobile devices, which suffers from high latency, lower throughput and intermittent poor connections. Various recent works proposed to use quantisation or sparsification techniques to reduce the amount of data that need to be communicated, for instance Top-k sparsification. However, the Top-k sparsification is lack of the cooperation between nodes. In this paper, we present the permutation mask (Pmask), a novel gradient sparsification technique for distributed model training. To unite nodes and improve the convergence rate, Pmask employs non-zero average and combined gradients. The corresponding convergence property is also given. We have applied permutation mask to image classification task on Cifar10, mini-ImageNet and SVHN, and the experiments show that Pmask leads to a faster distributed training and improves the accuracy.