Mitigating Stragglers in Distributed Learning: A Novel Framework with Gradient Coding
Jiale Wang, Jiahong Ning, Zechen He, Jian ping Zhao, Tingting Yang · 2024
The performance of distributed learning in multi-access edge computing networks is hindered by slow convergence caused by unstable communication links, heterogeneity among client devices, and stochastic fluctuations in computational power. This paper analyzes the types and causes of dropouts, considering both device heterogeneity and link instability. To address the challenges posed by stragglers, we propose a novel framework for distributed learning that does not rely on prior information about connectivity or dataset sharing. Our approach employs gradient sharing and coding strategies to mitigate the impact of stragglers, reducing computational overhead and enabling dynamic adjustments based on the availability and capabilities of edge nodes. Additionally, we develop a runtime model that incorporates communication and computational delays, addressing critical challenges such as link outages, timeout thresholds, and computational heterogeneity. The proposed framework provides a robust foundation for designing adaptive gradient coding techniques. An artificial delay is introduced to evaluate the effectiveness of various coding schemes under high-latency conditions. Experimental results demonstrate that the proposed partial coding strategies significantly reduce communication rounds and improve model accuracy compared to baseline methods, proving effective on both real-world and synthetic datasets.