MPI Allgather Utilizing CXL Shared Memory Pool in Multi-Node Computing Systems
Hooyoung Ahn, Seon Young Kim, Yoo-Mi Park, Woojong Han, Shinyoung Ahn, Tu Tran, Bharath Ramesh, Hari Subramoni, Dhabaleswar K. DK Panda · 2024
In Artificial Intelligence (AI) and high-performance computing (HPC), growing data and model sizes require distributed processing across multiple nodes due to single-node limitations, increasing inter-node communication. To address these challenges, we propose a novel MPI allgather method leveraging CXL technology, which supports composable architectures and dynamic resource allocation in data centers and HPC systems. Notably, CXL 3.1 facilitates cache coherence among nodes. The proposed allgather method uses the CXL shared memory pool as a communication buffer, outperforming existing algorithms for two reasons: First, CXL provides lower latency than Ethernet and IB, and second, by using the CXL shared memory pool as a shared communication buffer across multiple nodes, it significantly reduces the number of communications. To the best of our knowledge, this work is the first to explore combining MPI collective communication with CXL technology to optimize MPI allgather. Our proposed allgather method significantly reduces communication latency compared to traditional allgather methods by up to 42.14x, with a minimum improvement of 2.91x, as measured using the OSU Micro-Benchmark (OMB), a standard MPI benchmarking suite.