Runtime prediction of high-performance computing jobs based on ensemble learning
Xiaomeng Chen, Hui Zhang, Hanli Bai, Chunming Yang, Xujian Zhao, Bo Li · 2020
In high-performance computing job scheduling systems, to accurate predict the job runtime can effectively utilize the idle resource fragments generated during cluster computing as well as backfill them for improving scheduling performance. Because the runtime of high-performance computing jobs are affected by many factors thus are complicated non-linear problem. The ensemble machine learning method is used to predict the runtime of jobs in cluster computing. By comparing the prediction results of different models on the job log data sets from three real high-performance computing systems, it is found that the LightGBM algorithm has higher prediction accuracy, faster computation speed, shorter training time of the model and achieve better overall performance.