Enabling Efficient Use of MPI and PGAS Programming Models on Heterogeneous Clusters with High Performance Interconnects
Sreeram Potluri · OhioLink ETD Center (Ohio Library and Information Network) · 2014
Accelerators (such as NVIDIA GPUs) and coprocessors (such as Intel MIC/Xeon Phi) are fueling the growth of next-generation ultra-scale systems with high compute density and high performance per watt.However, they render these systems heterogeneous by introducing multiple levels of parallelism and varying computation/communication costs at each level.Application developers also use a hierarchy of programming models to extract maximum performance from these heterogeneous systems.They use models like CUDA, OpenCL, LEO+OpenMP, and others on an accelerator or a coprocessor while using higher level programming models like an MPI or a PGAS model across a cluster.Multiple programming models, their runtimes, and varying performance of communication at different levels of the system hierarchy have limited applications from the achieving peak performance on these systems.For example, in MPI and OpenSHMEM applications that run on GPU clusters, data is transferred between GPU and CPU using CUDA while it is exchanged between MPI processes running on different nodes using MPI or OpenSHMEM.This two-stage process for data movement introduces inefficiencies and hence limits performance.Communication in applications running on clusters with Intel MIC can happen over a myriad of channels depending on where the application processes are running: intra-MIC, intra-host, MIC-Host and MIC-MIC.Each of these channels has different performance characteristics.Runtimes have to be re-designed to optimize communication in such scenarios, while hiding system complexity from the user.Computation and communication overlap has been been a critical requirement for applications, to achieve peak performance on large-scale systems.Communication overheads have a magnified impact on heterogeneous clusters due to their higher compute density and hence, a higher wastage in compute power.Modern interconnects like InfiniBand, with their Remote DMA capabilities, enable asynchronous progress of communication, freeing up the cores to do useful computation.MPI and PGAS models offer light-weight, one-sided communication primitives that minimize process synchronization overheads and enable better computation and communication overlap.However, the design of one-sided communication on heterogeneous clusters is not well studied.Further, there has been limited literature to guide scientists in taking advantage of the one-sided communication semantics on high-end applications, more so on heterogeneous architectures.This dissertation has targeted several of these challenges for programming on GPU and Intel MIC clusters.Our work with MVAPICH2-GPU enabled the use of MPI in a unified manner, for communication from host and GPU device memories.It takes advantage of unified virtual addressing(UVA) provided by CUDA.We proposed designs in the MVAPICH2-GPU runtime to significantly improve the performance of internode and intranode GPU-GPU communication by pipelining and overlapping memory, PCIe and network transfers.We take advantage of CUDA features, such as IPC, GPUDirect RDMA, and CUDA kernels to further reduce communication overheads.MVAPICH2-GPU improves programmability by removing the need for developers to use CUDA and MPI for GPU-GPU communication, while improving performance through runtime-level optimizations that are transparent to the user.We have shown up to 69% and 45% improvement in point-to-point latency for data movement for 4Byte and 4MB messages, respectively.Likewise, the solutions improve the bandwidth by 2x and 56% for 4KByte and 64 KByte iii messages, respectively.Our work have been released as part of MVAPICH2 packages, making it the first MPI library to support direct GPU-GPU communication.It is currently deployed and used on several large GPU clusters across the world, including Tsubame 2.0 and Keeneland.We proposed novel extensions to the OpenSHMEM PGAS model that enable unified communication from host and GPU memories.We present designs for optimized internode and intranode one-sided communication on GPU clusters, using asynchronous threads and DMA-based techniques.The proposed extensions, coupled with an efficient runtime, improve the latency of 4 Byte shmem getmem latency by 90%, 40%, and 17%, for intra-IOH, inter-IOH, and inter-node GPU configurations with CUDA, respectively.They improve the performance of Stencil2D and BFS kernels by 65% and 12% on clusters of 192 and 96 GPUs, respectively.Through MVAPICH2-MIC, we proposed designs for an efficient MPI runtime on clusters with Intel Xeon Phi coprocessors.These designs improve performance of Intra-MIC, Intra-Node and Inter-Node communication on various cluster configurations, while hiding the system complexity from the user.Our designs take advantage of SCIF, Intel's low-level communication API, in addition to standard communication channels like shared memory and IB verbs, to offer substantial performance gains in performance of the MVAPICH2 MPI library.PRISM, a proxy-based multi-channel design in MVAPICH2-MIC allows applications to overcome the performance bottlenecks imposed by state-of-the-art processor architectures and extract the full compute potential of the MIC coprocessors.The proposed designs deliver up to 70% improvement in the point-to-point latency and more than 6x improvement in peak uni-directional bandwidth from Xeon Phi to the Host.Using PRISM, we improve inter-node latency between MICs by up to 65% and bandwidth by up to 5 times.it.I would like to take this opportunity to thank all of them.My family -my father, Dr. Gangadhara Rao, who has always given me inspiration and complete freedom to go after my dreams; my mother, Dr. Sarada