Reducing Communication Overhead via CEO in Distributed Training
Afshin Abdi, Faramarz Fekri · 2019
One of the main challenges in distributed training, especially in sensor networks, is the communication cost due to the transmission of the parameters of the model and synchronization across nodes, a.k.a. workers. We frame this problem as a central estimation officer (CEO) problem; starting from an initial guess for the model shared by all workers and the server, at each iteration of training, the workers refine the model parameters by training over their available dataset and send them to the server. The server merges the received information from the workers to get a better estimate of the optimum parameter of the model and sends it back to the workers. We propose using CEO framework to tackle the communication issue in distributed training. The proposed method consists of three major blocks: 1) dithered and nested quantization at the workers, 2) distributed source coding to incorporate the correlation among workers for further reduction in communication bit rate, and 3) decoding the data received from the workers and estimating the optimum parameters at the server. We have verified the effectiveness of our proposed approach via simulations. We have shown that our proposed method reduces the communication bit-rate compared to the other existing methods while it can achieve nearly the same convergence speed as of the baseline training (no quantization of the parameters) in our experiments.