Stochastic First-Order Methods Over Distributed Data
Muhammad I. Qureshi, Usman A. Khan · 2022
In this paper, we study the problem of learning from data available over a network of geographically distributed nodes. Each node possess a private local cost function and the goal is to minimize the global cost defined as the average of all local costs. Assuming that the cost functions are smooth and strongly-convex, and that the information exchange among the nodes can be asymmetric, we propose two first-order stochastic optimization methods PushSVRG and AB-SVRG converging to the global minimum. Both methods use net-work level gradient tracking to eliminate the dissimilarity among heterogeneous data distribution and node level vari-ance reduction to mitigate the variance caused by imper-fect (local) gradient information. To eliminate the asym-metry of information exchange caused by the communication, PushSVRG uses column-stochastic weights and push-sum consensus while AB-SVRG uses both row and column stochastic weights and does not require the extra push-sum iterations. We compare the proposed methods with related work on first-order stochastic optimization using extensive numerical experiments and highlight the practical aspects of different variance reduction techniques.