CTCCL: Cost-Efficient Joint Device-Network Load Balancing for LLM Training in RoCE-based Intelligent Computing Network
Zhuotong Li, Liang Xu, Ziqi Huang, Shuyun Qian, Hongwei Bu, Ming Chi Yang, Mengyun Luan, W Chen, Wen Xu · 2025
Pre-training large language models (LLMs) in data centers (DCs) is a complex yet essential task that requires vast computational resources and carefully designed infrastructures to enable efficient, large-scale distributed learning.However, without effective load balancing in RDMA over Converged Ethernet (RoCE) networks, network congestion and latency can create significant bottlenecks, disrupting data transmission, reducing resource utilization, and prolonging training times, ultimately compromising the scalability and performance of LLM training.To address these challenges, we propose and develop an innovative and cost-effective joint Device-Network Load Balancing (DNLB) approach.Built on our custom collective communication library, CTCCL, DNLB