Progress and Bandwidth Aware Partial Model Synchronization With Selective Reduce
Shouxi Luo, Zhen Liu, Qing Rao, Ke Li, Huanlai Xing · IEEE Transactions on Network Science and Engineering · 2025
By allowing partial instead of all workers to participate in a round of model synchronization, the recent proposal ofpartial reduceprovides a promising way to prevent the entire training from being blocked by straggler workers in heterogeneous data-parallel distributed training. However, its current designs are far from optimal, as it selects workers for partial model synchronization agnostic to both their training progress and available bandwidths. To address these issues, we analyze the design space and proposeselective reduce. By exploring the idea ofwaitingfor more workers to be ready and splitting them into partial synchronization groups, based on the state of their training progress and available bandwidths,selective reducecould not only enlarge the scale of synchronization (i.e., the number of involved workers) thus accelerating the convergence of the training but also reduce the time cost of synchronization thus making the training iterate faster. Extensive evaluations confirm thatselective reduceoutperformspartial reduceand is robust to both inaccurate bandwidth estimations and unknown-in-advance runtime distributions of training computation.