Performance Optimization of Allreduce Operation for Multi-GPU Systems
Akira Nukada · 2021 IEEE International Conference on Big Data (Big Data) · 2021
Allreduce is one of the important collective commu-nications used in distributed deep learning. We present a novel hybrid allreduce algorithm optimized for multi-GPU systems with NV-Link, which is current main computing platform for data-parallel distributed deep learning. The hybrid algorithm efficiently utilizes direct access to memory of peer GPU devices via the NV-Link. In addition to NV-Link, the hybrid algorithm employs PCI-Express network as extra bandwidth between GPU devices. To use the PCI-Express network we need to explicitly copy the data to host memory and these operations are software-pipelined. By selecting optimal parameters, the hybrid allreduce algorithm outperforms NVIDIA’s NCCL library.