An Empirical study of Gradient Compression Techniques for Federated Learning
Mradula Sharma, Parmeet Kaur · 2023
Federated Learning (FL) is a promising technique for decentralizing machine learning on multiple devices. In a FL environment, clients train an initial global ML model using their local data. They transfer the weights or gradients of the trained model to the server which, in turn, aggregates the received weights or gradients to obtain an improved global model. Thus, data is not shared with the server and this enhances user privacy as well as reduces the required communication bandwidth. However, resource-constrained devices and bandwidth-limited networks can fully leverage FL if the size of the weights or gradients can be reduced too. This paper performs an empirical investigation of a few existing gradient compression techniques for FL. The techniques of sparsification and Wyner-Ziv coding are studied and evaluated with respect to global model accuracy and communication overhead in various Federated and centralized scenarios. It is observed that the accuracy of the model is compromised if communication overhead is reduced. However, it is possible to achieve a balance between the two by selecting a compression gradient scheme according to the system requirements.