XMP-IO function and its application to MapReduce on the K computer
Tomotake Nakamura, Mitsuhisa Sato · Advances in parallel computing · 2014
In a large-scale parallel system, parallel I/O libraries, such as MPI-IO, are required to perform I/O operations between nodes and file systems. XcalableMP is a PGAS parallel programming language, which allows the programmer to define a distributed array over nodes to support typical data-parallel programming with global-view programming. In order to perform I/O operations of a distributed array efficiently, the information of a distributed array described in the program can be used for parallel I/O operations. XMP-IO is a parallel I/O API defined as a part of the XcalableMP specification. In this paper, we show that the productivity and performance of XMP-IO are nearly equal to those of MPI-IO. We verified that XMP-IO can be applied to the I/O of the MapReduce application. We evaluated the performance of these I/O on the K computer, which uses the Fujitsu Exabyte File System (FEFS), a high-speed parallel distributed processing file system FEFS.