POSTER: Leveraging deep memory hierarchies for data staging in coupled data-intensive simulation workflows
Tong Jin, Fan Zhang, Qian Sun, Hoang Bui, Norbert Podhorszki, Scott Klasky, Hemanth Kolla, Jacqueline M. Chen, Robert J. Hager, C. S. Chang, Manish Parashar · 2014
Next generation in-situ/in-transit data processing has been proposed for addressing data challenges at extreme scales. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with limited DRAM capacity in High End Computing clusters will impact the effectiveness of this approach. In this work, we propose using deep memory levels for data staging, utilizing a multi-tiered data staging method with both DRAM and solid state disk (SSD). This approach allows us to support both code coupling and data management for data intensive simulations in cluster environment. We also show how an application-aware data placement mechanism can dynamically manage and optimize data placement across DRAM and SSD storage levels in staging method. We present experimental results on Sith - an Infiniband cluster at Oak Ridge, and evaluate its performance using combustion (S3D) and fusion (XGC) simulations.