Exploiting Process Imbalance to Improve MPI Collective Operations in Hierarchical Systems
Benjamin S. Parsons, Vijay S. Pai · 2015
This work improves the performance of MPI collective communication operations in the presence of imbalanced process arrival times. High performance collective communications are crucial for the performance and scalability of applications, and imbalanced process arrival times are common in these applications. A micro-benchmark is used to investigate the nature of process imbalance with perfectly balanced workloads, and understand the nature of inter- versus intra-node imbalance. These insights are then used to develop imbalance-tolerant reduction, broadcast, and all-to-all algorithms, which minimize the synchronization delay observed by early arriving processes.