Silo & HDF5 I/O Scaling Improvements on BG/P Systems
Michael Collette, M. C. Miller · University of North Texas Digital Library (University of North Texas) · 2010
Silo and HDF5 are I/O libraries used by many codes important to the LLNL's Weapons and Complex Integration (WCI) mission. In the past year, modest adjustments and tuning of Silo, HDF5 and the I/O configuration of the BG/P platform, Dawn, were undertaken. A key goal of this work was to improve I/O performance without requiring any changes in the application codes themselves. In particular, the application codes have been allowed to continue to use a simplified yet highly flexible I/O paradigm known as 'Poor Man's Parallel I/O', where scalability is achieved through concurrent, serial I/O to multiple files. The results demonstrate substantial performance gains (better than 50x in many cases) at large scale (greater than 64,000 MPI tasks). They describe key enhancements made to Silo, HDF5 and the I/O configuration of our BG/P platform and present very favorable results from scalability studies over a wide range of operating scenarios.