Understanding Parallel I/O Performance and Tuning
Suren Byna · 2022
Performance of parallel I/O is critical for large-scale scientific applications to store and access data from parallel file systems on high-performance computing (HPC) systems. These applications use HPC systems often to generate and analyze large amounts of data. They use the parallel I/O software stack for accessing and retrieving data. This stack includes several layers of software libraries - high-level I/O libraries such as HDF5, middleware (MPI-IO), and low-level I/O libraries (POSIX, STD-IO). Each of these layers have complex inter-dependencies among them that impact the I/O performance significantly. As a result, scientific applications frequently spend a large fraction of their execution time in reading and writing data on parallel file systems. These inter-dependencies also complicate tuning parallel I/O performance.