DeSync: Proactive Congestion Control via Random Delay Offsets for Large-Scale ML Training
Xingbo Feng, Zhuyun Qi, Yi Wang, Ziyao Huang, Yan Liu, Jiashuo Lin, Chenxi Ling, Weichao Li, Jin Zhang, Jianping Wang · 2025
Synchronization-induced congestion is a critical performance bottleneck in modern distributed machine learning (ML) training, where simultaneous gradient exchanges create bursty traffic patterns. Existing solutions, both reactive and proactive, struggle to balance throughput and latency in the presence of synchronized flows. We propose DeSync, a proactive traffic shaping scheme that introduces structured random delay to de-synchronize communication rounds. Evaluations with DCQCN, HPCC, DCTCP, and TIMELY demonstrate that DeSync significantly improves FCT, job completion times, and congestion metrics, enhancing existing CC mechanisms without specialized hardware.