S 3 GD-MV: Sparse-SignSGD with Majority Vote for Communication-Efficient Distributed Learning
Chanho Park, Namyoon Lee · 2023
This paper presents S3GD-MV, a communication-efficient distributed learning algorithm that combines the benefits of sparsification and sign quantization. In S3GD-MV, each worker selects the top-K largest components of the local gradient vector in magnitude and only sends the signs of the selected components to the server, which aggregates the signs via a majority vote and returns the result. Our analysis shows that when the sparsification parameter is properly selected, S3GD-MV converges as quickly as signSGD for smooth non-convex functions, but with significantly reduced communication costs. Simulation results of training a convolutional neural network on the MNIST dataset show that S3GD-MV reduces communication costs compared to other conventional optimizers, while improving test accuracy.