High end scientific codes with computational I/O pipelines
Fang Zheng, Jianting Cao, Jai Dayal, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Hasan Abbasi, Scott Klasky, Norbert Podhorszki · 2011
This paper uses computational I/O pipelines to integrate computations into the I/O path that perform data analytics on the data generated by scientific simulations. A novel attribute is the use of a pluggable execution environment in which analysis tools can be orchestrated into a multi-stage pipeline for processing simulation output data. Performance considerations are addressed through the use of a high performance data transport. The approach is evaluated with the end-to-end performance of a Magnetohydrodynamics application at large scale.