Towards effective scheduling policies for many‐task applications: Practice and experience based on HTCaaS
Jik‐Soo Kim, Bui The Quang, Seungwoo Rho, Seoyoung C. Kim, Sangwan Kim, Vincent Breton, Soonwook Hwang · Concurrency and Computation Practice and Experience · 2017
Summary In this paper, we conduct a comparative study of relatively simple yet effective scheduling policies for many‐task applications where multiple users with varying numbers of tasks are actively sharing a common system infrastructure. We have implemented three different scheduling mechanisms that can address fairness and user response time respectively in a common middleware stack called HTCaaS, which is a pilot‐job–based multi‐level scheduling system running on top of production‐level clusters. As a representative case of our many‐task applications, we have leveraged the virtual screening application, which is a computational technique used in drug discovery process to select the most promising candidate drugs for in vitro testing from millions of chemical compounds, which typically requires a substantial amount of computing resources and efficient processing of docking simulations. Our comparative experimental results of different scheduling policies show how we can effectively support multiple users in a shared resource environment by balancing between user satisfaction and overall system performance, provide guidelines to improve system utilization, and address additional technical challenges to support various many‐task applications.