Reproducible MPI Micro-Benchmarking Isn't As Easy As You Think
Sascha Hunold, Alexandra Carpen-Amarie, Jesper Larsson Träff · 2014
The Message Passing Interface (MPI) is the prevalent programming model for supercomputers. Optimizing the performance of individual MPI functions is therefore of great interest for the HPC community. However, a fair comparison of different algorithms and implementations requires a statistically sound analysis. It is often overlooked that the time to complete an MPI communication function does not only depend on internal factors such as the algorithm but also on external factors such as the system noise. Most noise produced by the system is uncontrollable without changing the software stack, e.g., the memory allocation method used by the operating system. Possibly controllable factors have not yet been identified as such in this context. We investigate several possible factors---which have been discovered in other microbenchmarks---whether they have a significant effect on the execution time of MPI functions. We experimentally and statistically show that results obtained with other common benchmarking methods for MPI functions can be misleading when comparing alternatives. To overcome these issues, we explain how to carefully design MPI micro-benchmarking experiments and how to make a fair, statistically sound comparison of MPI implementations.