Dynamic Resource Management for Elastic Scientific Workflows using PMIx
Rajat Bhattarai, Howard Pritchard, Sheikh Ghafoor · 2024
In current scientific workflows, the computational needs of tasks might not be known when it is submitted to a system for execution. Current resource management (RM) systems and workflow managers (WFMs) provide limited support for dynamic resource allocation in HPC systems, thus the common approach is to request the maximum resources needed for a maximum time, potentially wasting resources. However, in some cases, maximum resources may not be estimated a priori, as a result, a workflow may be completed after the deadline, or in cases, the task may terminated by the resource manager. A combination of workflow manager and resource management system that can accommodate a fine-grain elastic resource allocation during the execution of a workflow would alleviate this problem. This paper presents a dynamic elastic resource management framework based on the Parsl workflow manager and PMIx-enabled SLURM and reports the early evaluation of the framework using two workflow applications.