Combining Static and Dynamic Storage Management for Data Intensive Scientific Workflows
Nicholas Hazekamp, Nathaniel Kremer-Herman, Benjamín Tovar, Haiyan Meng, Olivia Choudhury, Scott Emrich, Douglas Thain · IEEE Transactions on Parallel and Distributed Systems · 2017
Workflow management systems are widely used to express and execute highly parallel applications. For data-intensive workflows, storage can be the constraining resource: The number of tasks running at once must be artificially limited to not overflow the space available in the filesystem. It is all too easy for a user to dispatch a workflow which consumes all available storage and disrupts all system users. To address these issues, we present a three-tiered approach to workflow storage management: (1) A static analysis algorithm which analyzes the storage needs of a workflow before execution, giving a realistic prediction of success or failure. (2) An online storage management algorithm which accounts for the storage needed by future tasks to avoid deadlock at runtime. (3) A task containment system which limits storage consumption of individual tasks, enabling the strong guarantees of the static analysis and dynamic management algorithms. We demonstrate the application of these techniques on three complex workflows.