Aggregating Stochastic Gradients in Distributed Optimization
Thinh T. Doan · 2018
Motivated by broad applications in computer science and engineering, we study distributed algorithms for optimization problems over a network of nodes, where the goal is to optimize a global objective composed of a sum of local functions. In solving such problems, we propose an algorithm, namely, distributed aggregated stochastic gradient method, which only requires local computation and communication. Our main contribution is to show that the proposed algorithm achieves a linear convergence rate to the neighborhood of the problem solution. In particular, we study the rate of convergence of the algorithm without requiring the Lipschitz continuity of local functions, as often assumed in distributed stochastic gradient methods. In addition, we provide simulations to show that our method outperforms distributed stochastic gradient methods in solving the important linear regression problems over networks.