Communication-Efficient Quantized SGD for Learning Polynomial Neural Network
Zhanpeng Yang, Yong Zhou, Youlong Wu, Yuanming Shi · 2021
This paper establishes the convergence rates for fitting a polynomial neural network with quadratic activation function via the mini-batch Stochastic Gradient Descent (SGD) algorithm. Specifically, we focus on the parallel implementation of calculating mini-batch gradients on a distributed computing platform. We first illustrate that the SGD converges at a linear rate to the optimal solution, and the convergence rate can be characterized as a function of mini-batch sizes. Next, we deploy the SGD with a distributed approach across multiple processors, where the partial mini-batch gradient is calculated and quantized to send to a master processor in each iteration, yielding a Quantized Stochastic Gradient Descent (QSGD) algorithm. This scheme can effectively reduce the communication overhead by the quantization strategy. Furthermore, we reveal that QSGD provably maintains a similar convergence rate of SGD to a globally optimal solution while significantly reduces the communication cost. In particular, the number of bits required for quantization and the mini-batch size affect the convergence rate of QSGD.