Timely offloading of result-data in HPC centers
Henry M. Monti, Ali Raza Butt, Sudharshan S. Vazhkudai · 2008
High performance computing is facing an exponential growth in job output dataset sizes. This implies a significant commitment of supercomputing center resources---most notably, precious scratch space---in handling data staging and offloading. However, the scratch area is typically managed using simple "purge policies", without sophisticated "end-user data services" that are required to balance center's resource consumption and user serviceability. End-user data services such as offloading are performed using point-to-point transfers that are unable to reconcile center's purge and users delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant.