IOScout: an I/O Characteristics Prediction Method for the Supercomputer Jobs
Yuqi Li, Liquan Xiao, Jinghua Feng, Jian Zhang, Gang Zheng, Yuan Yuan · 2023
Modern exascale supercomputers require more efficient I/O service than traditional single-shared filesystems can provide to support applications with varying I/O loads. Although current supercomputers can offer multiple storage resources for meeting different job I/O requirements, mainstream job schedulers need the ability to allocate hardware based on job I/O characteristics automatically. Job schedulers must first predict the I/O characteristics of the high-performance computing job to enable this ability. However, the traditional I/O feature prediction method uses I/O performance metrics collected after the job starts. The I/O channels are generally built for the job at the beginning, meaning the job schedulers must predict I/O characteristics before the job starts. This paper proposes an I/O characteristics prediction method for supercomputer jobs using only job description information, which can be collected before the jobs start and contain text and numerical data. We solved the problem of integrating different data types and selected the most suitable model through model selectors during the data training process. The method achieves 80.2% accuracy and 88.6% precision through validation using more than 40 days of job records on the new generation Tianhe supercomputer.