IWFMS: An Internal Workflow Management System/Optimizer for Hadoop
Lian Liu, Yao Shen · 2014
The scale of jobs running on parallel computation platform such as Hadoop is increasing quickly, thus workflow engines that manage data processing jobs have become increasingly important. However, traditional workflow management systems are mostly outsiders to Hadoop and cannot fulfill many important requirements, such as user constraints and scheduling optimizations. In this paper, we present IWFMS, an internal workflow scheduling system/optimizer that 1) Manage the resource allocation and execution of jobs in workflows to achieve higher efficiency, 2) Schedule workflows to meet a much richer set of user constraints such as deadlines, priorities and workflow trigger events. We discuss the architecture of its key components and evaluate its features and performance.