Communication cost reduction using sparse ternary compression and encoding for FedAvg
Thi Quynh Khanh Dinh, Thanh-Hai Tran, Thi‐Lan Le · 2021 International Conference on Information and Communication Technology Convergence (ICTC) · 2021
Nowadays, Federated Learning (FL), a training paradigm in which data are stored locally and used to train model on client devices has emerged thanks to the growing computational power of client devices as well as the concern about transmitting private information. In FL, multiple parties jointly train a model with their local dataset. By this way, privacy of clients' data is kept. However, FL has to face to communication cost during training as the model and model weights have to be sent to server to update the global model. Several methods have been proposed to address this issue. In this work, we propose a model weight compression and encoding during model uploading for Federated Averaging (FedAvg) - a widely used framework in FL. Our weight compression is inspired by Sparse Ternary Compression algorithm with a modification to be applicable to FedAvg. We also utilize compressed weights' characteristics to encode them hence the communication cost can be reduced. The experimental results on an image classification task with MNIST dataset demonstrate that our method is able to reduce the communication cost without considerably worsening the model accuracy.