Calibration and predictive modeling of computer systems
J. Craig Lowery · 1992
As the workload on a computer system increases, its performance degrades. The manager of a system is usually presented with several possible solutions (i.e., upgrades), each with some associated cost. To determine the best performance/cost alternative, predictive models may be used. Predictive modeling is a technique in which an abstract model of the computer system is constructed. Proposed changes are made to the model, not the actual system. This provides a cost effective means of improving the probability of choosing the best option without having to actually implement all options. However, since every model is, at some level, an approximation of the actual system, errors occur. Errors are introduced when (1) assumptions are made to account for the unknown, (2) detailed data is generalized, or (3) the workload characterization is inaccurate. Calibration is a procedure by which an inaccurate baseline model is forced to match the actual system in its pre-upgrade configuration. There are several ways in which a model can be calibrated, and the choice affects the model's predictive accuracy. Historically, the choice of calibration is an ad hoc one. This thesis approaches the subject of calibration in a structured manner. The scientific method is applied to the study of calibration in the context of product form queueing network models. An overview of predictive modeling and the role of calibration is given. The relationship between calibration, approximation, and sensitivity analysis is presented. A classification of calibrations is developed and used as a framework for case studies. Case studies of five computer systems are conducted. The results of these empirical studies are analyzed and calibration heuristics (i.e., rules of thumb) are found and justified. These heuristics are useful in the a priori determination of an appropriate calibration to use in a given modeling context. It is also shown that exact analysis can be performed in restricted cases. Calibration in the context of Petri net models is also addressed. This thesis demonstrates the value of calibration using a structured scientific approach.