Megatuner: An Offline DCQCN Parameters Tuner For Large-scale Models
Xin Qi, Xiaoxiang Wang, He Liu, Fangzheng Jiao, Xiaohe Hu, Yang Jing · 2024
As high-performance computing clusters (HPCC) scale up and single-node computing power improves, the bottle-neck of large-scale models training is shifting from computation to network communication. Datacenter Quantized Congestion Notification (DCQCN), as the most widely used congestion control algorithm in lossless Remote Direct Memory Access (RDMA) networks today, provides HPCC with low-latency, high-bandwidth, and high-quality network services. However, DCQCN has more than ten adjustable parameters on RDMA-enabled network inter-face cards (RNICs) and switches, and currently, relying on expert experience to adjust these parameters in engineering is inefficient. Additionally, even minor increases in single-iteration time during training can lead to significant resource wastage. Therefore, after conducting an in-depth analysis of different heuristic search algorithms and loss functions, we propose Megatuner—an offline DCQCN parameters tuner tailored for the traffic pattern of large-scale models. We conducted traffic simulations on the collective communications at the foundation of large-scale models, testing both on simulation platforms and in real environments. The results show that parameters tuned by Megatuner significantly outperform those set based on expert experience. We also addresses the issue of bandwidth degradation in real environments, ensuring long-term optimization of network performance.