Cloud-native workflow scheduling using a hybrid priority rule and dynamic task parallelism
Jungeun Shin, Diana Arroyo, Asser N. Tantawi, Chen Wang, Alaa Youssef, Rakesh Nagi · 2022
Demand for efficient cloud-native workflow scheduling is growing as many data science workloads are composed of several tasks with dependencies. As container technology becomes more prevalent in cloud communities, containerized workflow orchestration tools are introduced and become standard for scheduling workflows. However, current schedulers use simple heuristics and rely on the user's choice on priority and parallelism level of tasks without accounting for workflow-specific information.