Experimental evaluation of infiniband transport over local- and wide-area networks
Stephen Ross Carter, Makia Minich, Nageswara S. V. Rao · Spring Simulation Multiconference · 2007
High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.