Investigating Data Movement Strategies for Distribution of Repartitioned Data
John‐Paul Robinson, Ke Fan, Steve Petruzza, Thomas Gilray, Sidharth Kumar · 2024
Repartitioning in a parallel setting can be defined as the task of redistributing data across processes based on a newly imposed grid/layout. Repartitioning is a fundamental problem, with applications in domains that typically involve computation on tiles (blocks/patches) of varying resolution, for example, while creating multi-resolution data formats in in situ mode (such as the JPEG format and its variants). This paper explores the performance and tradeoffs of different ways to perform the data redistribution phase. In particular, we explore a greedy scheme that aims to minimize data movement while compromising on load balancing and a balanced scheme that aims to create a balanced load across processes while compromising on data movement. For both these schemes, we measure the impact of buffer size on MPI point-to-point communication performance when using two different communication patterns: a per-patch (staggered data transfer) and a per-rank (aggregated data transfer). Our experimental study finds that the reduced data movement of the greedy scheme leads to reduced transfer times during redistribution. Furthermore, we conclude that the per-patch communication pattern outperforms per-rank communication.