Accelerated Primal-Dual Algorithm for Distributed Non-convex Optimization
Shengjun Zhang, Colleen P. Bailey · 2021 IEEE Symposium Series on Computational Intelligence (SSCI) · 2021
This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed stochastic gradient primal-dual with fixed parameters and “powerball” (DSGPA-F-PH) method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate$\mathcal{O}(1/\sqrt{nT})$for general smooth (possi-bly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare with state-of-the-art distributed stochastic gradient descent (SGD) algorithms and centralized SGD algorithms.