Efficient Inter-Datacenter AllReduce With Multiple Trees
Shouxi Luo, Renyi Wang, Huanlai Xing · IEEE Transactions on Network Science and Engineering · 2024
In this paper, we look into the problem of achieving efficient inter-datacenter AllReduce operations for geo-distributed machine learning (Geo-DML). Compared with intra-datacenter distributed training, the heterogeneous wide-area network (WAN) connections among Geo-DML workers are scarce, expensive, and unstable, making existing proposals designed for homogeneous networks fall short. Despite that some recent optimizations have been proposed for Geo-DML, they break the consistency semantics of bulk synchronous parallel (BSP), thus bringing no benefit to the widely existing BSP-based applications. To address these issues, we proposemTree, a topology management suite for Geo-DML. With the global view of the heterogeneous WAN connections,mTreebuilds multiple optimized spanning trees along with suggested workload distribution proportions, respecting the constraints of both the number of trees and their maximum height specified by the training. Based on these results, geo-distributed workers could launch concurrent tree-based pipelined AllReduce operations to make efficient use of the heterogeneous network. Detailed performance studies on real-world network topologies imply thatmTreeachieves efficient AllReduce, significantly outperforming existing solutions.