Leveraging One-Sided Communication for Sparse Triangular Solvers
Nan Ding, Samuel Williams, Liu Yang, Xiaoye Sherry Li · Society for Industrial and Applied Mathematics eBooks · 2020
In this paper, we implement and evaluate a one-sided communication-based distributed-memory sparse triangular solve (SpTRSV). SpTRSV is used in conjunction with Sparse LU to affect preconditioning in linear solvers. One-sided communication paradigms enjoy higher effective network bandwidth and lower synchronization costs compared to their two-sided counterparts. We use a passive target mode in one-sided communication to implement a synchronization-free task queue to manage the messaging between producer-consumer pairs. Whereas some numerical methods lend themselves to simple performance analysis, the DAG-based computational graph of SpTRSV demands we construct a critical path performance model in order to assess our observed performance relative to machine capabilities. In alignment with our model, our foMPI-based one-sided implementation of SpTRSV reduces communication time by 1.5× to 2.5× and improves SpTRSV solver performance by up to 2.4× compared to SuperLU-DIST's two-sided MPI implementation running on 64 to 4,096 processes on Cray supercomputers.