Load Imbalance in HPC Applications: Improved Profiling and New Ways to Use Wasted Cycles
Shining Yang, Xiteng Yao, Grace Nansamba, Amr Akmal Abouelmagd, Anthony Skjellum, Martin Herbordt · 2025
As the scale of High Performance Computing (HPC) applications increases, load imbalance and other inefficiencies become correspondingly more significant. In this work we extend a previously described system that reclaims idle computation time during MPI synchronization. We begin by applying a new profiler to a new target system (Stampede III) and observing application characteristics. We find that for miniFE there is immense potential for capturing wasted cycles (roughly 30%). Moreover these cycles do not appear to be caused systematically by particular ranks, but rather by a number of ranks that vary without obvious pattern. We then propose a new runtime algorithm that improves CPU cycle utilization by an additional 10–15% over the previous best result.