Granularity- and Interference-Aware GPU Sharing with MPS
Alex Weaver, Krishna M. Kavi, Dejan S. Milojicic, Rolando P. Hong Enriquez, Ninad Hogade, Alok Mishra, Gayatri Mehta · 2024
GPU acceleration has become central to achieving exascale performance for high-performance computing (HPC) applications. Even at this extreme scale, most scientific and HPC-scale DNN applications under-utilize GPU resources. Existing NVIDIA GPU sharing mechanisms—namely, CUDA Multi-Process Service (MPS)—can be used to increase utilization, throughput, and energy efficiency. However, naively co-scheduling workflows often does not yield optimal results. Scheduling multiple workloads with high resource utilization on the same set of GPUs, for example, leads to performance degradation due to resource contention. In short, GPU sharing must be granularity- and interference-aware to maximize the benefit of co-scheduling. We propose a scheduling approach that optimizes workflow scheduling configurations to maximize given system metrics–i.e., throughput and energy efficiency; uses workload profiling data to right-size GPU resources for combinations of HPC workflows; and collocates workflows using existing concurrency/sharing mechanisms. We show that choosing the right arrangement of workflows to collocate can increase throughput by as much as 2x and energy efficiency by 1.6x using CUDA MPS on NVIDIA A100X GPUs.