ESync: Accelerating Intra-Domain Federated Learning in Heterogeneous Data Centers
Zonghang Li, Huaman Zhou, Tianyao Zhou, Hongfang Yu, Zenglin Xu, Gang Sun · IEEE Transactions on Services Computing · 2020
Federated Learning (FL) serves privacy-preserving collaborative learning among multiple isolated parties, while retaining their privacy data locally. Cross-device and cross-silo FL have achieved great success in cross-domain applications, in which the scarce communication resource is the primary bottleneck. Driven by the need to combine heterogeneous machines from different parties to build a shared data center, we foundintra-domain FL, a new type of FL in which isolated parties collaborate in the shared data center, and strong computational heterogeneity becomes the primary bottleneck. To mitigate the training inefficiency caused by stragglers, this article proposes an efficient synchronization algorithmESync, which allows parties to train different iterations locally under the coordination of a novel schedulerState Server. We give the boundaries of weight divergence and optimality gap ofESync, and analyze the trade-off between convergence accuracy and communication efficiency. Extensive experiments are conducted to compareESyncwith SSGD, ASGD, DC-ASGD, FedAvg, FedAsync, TiFL, and FedDrop under strong computational heterogeneity. Numerical results show thatESyncachieves great speed up without loss of accuracy, and therefore demonstrate the effectiveness ofESyncin both training efficiency and converged accuracy.