Native Data Representation: an Efficient Wire Format for High Performance Computing
Greg Eisenhauer, Fabián E. Bustamante, Karsten Schwan · 2001
Flexible and high-performance data exchange is becoming increasingly important. This trend is due in part to the growing interest among highperformance researchers in tool- and component-based approaches to software development. In trying to reap the well-known benets of these approaches, the question of what communications infrastructure should be used to link the various application components arises. Traditional HPC-style communication libraries such as MPI oer good performance, but are not intended for loosely-coupled systems. Object- and metadatabased approaches like XML oer the needed plug-and-play exibility, but with signicantly lower performance. We observe that the exibility and baseline performance of data exchange systems are strongly determined by their `wire formats', or by how they represent data for transmission in the heterogeneous environments. Upon examining the performance implications of using a number of dierent wire formats, we propose an alternative approach to exible high-performance data exchange, Native Data Representation, and evaluate its current implementation in the Portable Binary I/O library. 1