A Family of Distributed Momentum Methods Over Directed Graphs With Linear Convergence
Juan Gao, Xinwei Liu, Yu‐Hong Dai, Yakui Huang, Peng Yang · IEEE Transactions on Automatic Control · 2022
We consider the distributed optimization to minimize a sum of smooth and strongly convex local objective functions over directed graphs. Using row- and column-stochastic weights, we propose a family of distributed momentum methods. It is a parametric distributed momentum (PDM) method, for which different values of parameter can lead to different distributed momentum methods. PDM includes the distributed heavy-ball method ($\mathcal {AB}m$) and the distributed Nesterov gradient method ($\mathcal {ABN}$) as its special cases. When the step sizes and the momentum coefficient do not exceed some upper bounds, we prove that PDM can converge to the optimal solution at a global$R$-linear rate. The convergence result of PDM not only covers that of$\mathcal {AB}m$, but also supplements that of$\mathcal {ABN}$, which lacks theoretical convergence result. Simulation results on classification problems that arise in machine learning show that PDM with an appropriate negative parameter value can achieve faster acceleration than the existing distributed momentum algorithms.