Job Runtime Prediction Based on the Fusion of Voting Classification and Stacking Model
Bing Yan, Xubo Kong, Gan Zhou, Rong Dai · 2024
In distributed computing environments, many users have an uncertain understanding of job runtimes, often requesting more time than necessary when submitting jobs. This tendency results in low resource backfill efficiency due to inaccurate user estimations of job runtime. The extended queuing times for jobs, coupled with the inability to schedule available free resources promptly, lead to the waste of these resources, negatively impacting cluster utilization and user experience. Therefore, accurate prediction of job runtime is crucial for enhancing job scheduling efficiency. In this paper, we introduce a novel job runtime prediction method: historical job data are categorized by the length of job runtime, and the final classification prediction is generated using the Soft Voting classifier within the Voting integrated learning algorithm. This method leverages the prediction probabilities from three classification models-Random Forest, Decision Tree, and Support Vector Machine. Based on these classification predictions, we identify similar job sets for test jobs within the categorized sets and then predict job runtime using the Stacking model fusion technique. Experiments were conducted using historical datasets from two large-scale computing centers, with the Last-2 algorithm, LSH-sim algorithm, and SU framework algorithm selected for comparison. The experimental results demonstrate that, compared to these three baseline algorithms, our algorithm improves the average prediction accuracy by 25.0%, 7.5%, and 24.9%, respectively. It also reduces the average underestimation rate by 25%, 16.7%, and 7.7%, respectively. Thus, the algorithm proposed in this paper can effectively enhance prediction accuracy and decrease the underestimation rate.