Feasibility Study of Effective Remote I/O Using a Parallel NetCDF Interface in a Long-Latency Network
Yuichi Tsujita · 2008
Abstract—NetCDF provides portable and selfdescribing I/O data format for array-oriented data in scientific computation domains. Its parallel I/O interface named parallel netCDF (hereafter PnetCDF) provides parallel I/O operations with the help of an MPI interface. To realize such operations among computers which have different MPI libraries through a PnetCDF interface, a Stampi library was introduced as an underlying MPI library. Decomposition in multi-dimensional data leads to complex I/O operations in non-contiguous parallel I/O pattern with the help of a derived data type. However, times for data communications between computers are independent of the complexity because the transfered data are contiguous in memory buffer. Although it succeeded in effective remote I/O operations in a LAN environment, throughput is degraded due to unoptimized configuration for TCP sockets if interconnection among computers has long latency like WAN. We have addressed the degradation and we observed that applying an appropriate size in a socket buffer minimized I/O times effectively.