Comprehensive Analysis of Computational and I/O Resource Patterns Across Queues
Huang Bin, Gang Xu · 2024
In the current computing landscape, with the explosive growth of computing tasks, effective job resource utilization is of utmost importance for enhancing system performance. This study conducts a comprehensive analysis of job resource usage, focusing on the running time of job queues, core and memory resource utilization, and the characteristic distribution of job read-write I/O. Using a dataset containing 51,987 jobs from the Gaia Cluster over 3 months, we explore various aspects. The running time analysis reveals that different queues have distinct job runtime distributions. For instance, the interactive queue has a significant proportion of short-runtime jobs (30%-40%) for rapid data operations and long-runtime jobs (30%-40%) for complex tasks. The normal queue is dominated by medium-runtime jobs (50%-60%) to balance resource utilization. Core resource usage shows a trend where medium-core jobs are predominant (60%-80%) across queues due to factors like task complexity and cost-effectiveness. Memory usage is closely related to runtime, with medium-memory jobs corresponding to medium runtime and large-memory jobs having longer runtimes. In terms of I/O, the interactive queue has a high proportion of read-intensive and write-intensive jobs, while the normal queue is mainly write-intensive. These findings provide a detailed understanding of job resource usage patterns and offer practical guidance for optimizing computing systems.