ElastiQuant
Bharath Sudharsan, John G. Breslin, Muhammad Intizar Ali, Peter Corcoran, Rajiv Kumar Ranjan · Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing · 2022
In training distributed machine learning, communicating model updates among workers has always been a bottleneck. The magnitude of impact on the quality of resultant models is higher when distributed training on low hardware specification devices and in uncertain real-world IoT networks where congestion, latency, bandwidth issues are common. In this scenario, gradient quantization plus encoding is an effective way to reduce cost when communicating model updates. Other approaches can be to limit the client-server communication frequency, adaptive compression by varying the spacing between quantization levels, reusing outdated gradients, deep compression to reduce transmission packet size, and adaptive tuning of the number of bits transmitted per round. The optimization levels provided by such and other non-comprehensive approaches do not suffice for high-dimensional NN models with large size model updates.