Communication-Efficient Collaborative Learning of Geo-Distributed JointCloud from Heterogeneous Datasets
Xiaoli Li, Nan Liu, Chuan Chen, Zibin Zheng, Huizhong Li, Yan Qiang · 2020
With the popularity of cloud computing, the use of services provided by the cloud is increasing. Due to the high communication cost, and data privacy issues, a new crosscloud collaborative computing model is demanded instead of a single giant cloud. Coping with federated learning and Joint-Cloud, we propose a federated learning-based collaborative learning framework, in which the distributed cloud entities are able to learn the same model collaboratively. As compared to traditional cloud-centric approaches, the framework for JointCloud can reduce the network bandwidth overhead and guarantee privacy. However, there are two crucial challenges in the federated manner: heterogeneity and high communication overhead. To address the heterogeneity, we propose a Teacher-Student mechanism, the key of which is a regularization term incorporated with the objective function so as to adjust the gradients from the clients among JointCloud with different data distribution. Then, based on the Teacher-Student mechanism, we further present a communication-efficient federated optimation approach via joint Identification-Verification to reduce the communication rounds. We conduct extensive experiments on Non-IID datasets. The experimental results demonstrate that the proposed framework can significantly reduce communication costs and improve performance.