Variance Reduced Stochastic Optimization Over Directed Graphs with Row and Column Stochastic Weights
Muhammad I. Qureshi, Ran Xin, Soummya Kar, Usman A. Khan · 2023
This paper proposes AB-SAGA, a first-order distributed stochastic optimization method to minimize a finite sum of smooth and strongly convex functions distributed over an arbitrary directed graph. AB-SAGA removes the uncertainty caused by the stochastic gradients using node-level variance reduction and subsequently employs network-level gradient tracking to address the data dissimilarity across the nodes. Unlike existing methods that use the nonlinear push-sum correction to cancel the imbalance caused by the directed communication, the consensus updates in AB-SAGA are linear and use both row and column stochastic weights. We show that for a constant stepsize, AB-SAGA converges linearly to the global optimal. We quantify the directed nature of the underlying graph using an explicit directivity constant and characterize the regimes in which AB-SAGA achieves a linear speed-up over its centralized counterpart. Numerical experiments illustrate the convergence of AB-SAGA for strongly convex and non-convex problems.