An Efficient Inter-Node Communication System with Lightweight-Thread Scheduling
Takuya Fukuoka, Wataru Endo, Kenjiro Taura · 2019
In the era of multi-/many-core processors, there are increasing needs for middleware of high-performance computing to exploit both inter-node and intra-node parallelism. To overlap communication and computation efficiently, many studies have focused on MPI+ULT, a combination of MPI for inter-node parallelism and user-level threads (ULTs) for intra-node parallelism. However, there are mainly two problems in the existing MPI+ULT implementations. First, the use of MPI_THREAD_MULTIPLE to invoke MPI functions from multiple threads causes a performance bottleneck. Second, some MPI+ULT systems focus on the use of non-blocking communication and programmers have to manage both the start and the end of communication explicitly. To solve these problems, we introduce a high-performance MPI+ULT implementation MPI+myth. MPI+myth focuses on implicit overlapping of communication and computation without any code modifications to the applications. Furthermore, it can avoid the overhead of multi-threaded MPI invocations using a communication dedicated thread and adopts a new scheduling technique which achieves efficient load balancing by avoiding a situation that a core is occupied by blocking ULTs. In the evaluation, we demonstrate significant performance improvement compared with the existing hybrid programming methods using several microbenchmarks and one mini application miniFE. In addition, we illustrate that MPI+myth has the potential to overlap communication and computation and our new ULT scheduling technique can achieve load balancing more efficiently than existing ULT scheduling techniques.