Decentralized Stochastic Projection-Free Learning with Compressed Push-Sum
Robin Francis, Sundeep Prabhakar Chepuri · 2023
We consider decentralized stochastic learning methods with data being distributed among multiple nodes. The nodes communicate gradients with their connected neighbors. To reduce communication overhead, several compression techniques have been proposed for solving unconstrained optimization problems in a decentralized setting. In this paper, we focus on decentralized stochastic learning with convex constraints. Specifically, we propose a novel communication-efficient decentralized stochastic Frank-Wolfe algorithm to solve finite-sum constrained minimization problems by communicating only a compressed version of the gradient. The proposed method guarantees a convergence rate of about $O(n^{-1/2}k^{-1/3})$ for convex objectives, and for non-convex objectives, it guarantees a convergence rate of about $O(n^{-1/2}k^{-2/9})$, where n is the number of nodes in the network. We empirically validate the efficacy of the proposed algorithm in terms of communication overhead and suboptimality gap on several benchmark machine learning tasks.