AFNFA: An Approach to Automate NCCL Configuration Exploration
Zibo Wang, Yuhang Zhou, Chen Tian, Xiaoliang Wang, Xianping Chen · 2023
With the continuously increasing scale of deep neural network models, there is a clear trend towards distributed DNN model training. State-of-the-art training frameworks support this approach using collective communication libraries such as NCCL, MPI, Gloo, and Horovod. These libraries have many parameters that can be adjusted to fit different hardware environments, and these parameters can greatly impact training performance. Therefore, careful tuning of parameters for each training environment is required. However, given the large parameter space, manual exploration can be time-consuming and laborious.