QoS Support for Scientific Workflows Using Software-Defined Storage Resource Enclaves
Suman Karki, Bao Nguyen, Xuechen Zhang · 2018
Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes, executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many "stages" of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in longer than expected end-to-end I/O latency. The causes of such performance issues are missing a performance guarantee (e.g., lower bounds of I/O throughput) across stages of I/O pipelines and across layers of the I/O stacks. In this paper, we present the design and implementation of a novel data management infrastructure called Software-defined Storage Resource Enclaves (SIREN) at system levels to enforce end-to-end policies that dictate an I/O pipeline's performance. Our results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers while maintaining high system scalability and resource utilization.