A Simulation for Forecasting Compute Resource Usage
Saurabh Singh Adhikari, CHRISTIAN PLEWNIA, Chayakorn Netramai, Horst Lichter · 2021
The usage of compute resources by data processing jobs may change over time, requiring careful resource planning when an organization operates these resources itself in an on-premise private cloud. Ideally, the currently available resources always match the need of jobs executed on them. This way the resources would neither be overutilized, which is usually undesirable as the jobs might take longer, nor underutilized, which causes unnecessary costs for unused resources. When an organization decides to extend its private cloud resources, it can still take months until the servers are bought, delivered, and installed. Thus, the resources have to be planned carefully in advance. Estimating the future resource needs is difficult and influenced by many factors. In our experience, creating the estimate is often a manual process supported by self-designed spreadsheets; these spreadsheets are maintained by a single person from time to time and might even be replaced completely if someone else assumes that person's responsibility. However, this approach does not lead to transparent and verifiable forecasts that enable collaboration and learning from past decisions.