Towards Improving Resource Allocation for Multi-Tenant HPC Systems: An Exploratory HPC Cluster Utilization Case Study
Robert R. Kessler, Simon Volpert, Stefan Wesner · 2024
On-premise HPC systems are usually operated in such a way that the resources requested by its users are allocated exclusively to the corresponding jobs for the duration of their runtime. In order to guarantee the high availability of the resources and to minimize mutual interference due to the noisy-neighbor effect, resources are generally not overbooked, although this may lead to an under-utilization. In this context, we distinguish between two types of under-utilization: those relating to the allocation and those relating to the resources itself. Allocation under-utilization is a circumstance where not all available resources are being allocated, despite the fact that jobs are queued, due to potential cluster policies or bad scheduling. Resource under-utilization thereby mainly arises from bad performing code or due to task dependencies and is caused by the user. For both types of under-utilization, we define methodologies to determine under-utilization at the node and thus also at the cluster level. We apply our methodology to the MIT Supercloud dataset and evaluate it in terms of its cluster-wide utilization, due to the lack of specific data about resource allocation at the node level. We observed recurring longer phases of under-utilization in terms of both the allocation and the actual physical resource usage of the compute cores. In particular, within a 3-day time window around the peak allocation utilization of the overall cluster, we found a rather large disparity between the allocation and utilization of the computational resources of 14.7% with respect to the total available resource or 46.8% relative to the available allocated resources. We see major deficiencies with regard to the sustainable operation of such clusters, both in terms of the allocation under-utilization and the disparity between the actual resource utilization and the available allocated resources.