Decentralized Learning Based on Gradient Coding With Compressed Communication
Chengxi Li, Mikael Skoglund · IEEE Transactions on Signal Processing · 2024
This paper considers the problem ofdecentralized learning (DEL)with stragglers under the communication bottleneck. In the literature, various gradient coding techniques have been proposed fordistributed learningwith stragglers by letting the devices transmit encoded gradients based on redundant training data. However, those techniques can not be directly applied to fully decentralized scenarios as considered in this paper due to the lack of a global model in DEL. To overcome this shortcoming, we first propose a newgossip-based DEL method with gradientcoding (GOCO). In GOCO, to mitigate the negative impact of stragglers, the devices update the parameter vectors with encoded gradients based on stochastic gradient coding before averaging in a gossip-based manner. To further reduce the communication overhead associated with GOCO, we propose an enhanced version of GOCO, namely GOCO with compressed communication (2-GOCO), where the devices transmit compressed messages instead of the raw parameter vectors. The convergence of the proposed methods is analyzed for strongly convex loss functions. Simulation results demonstrate that the proposed methods outperform the baseline methods, which attain better learning performance under the same communication overhead.