A Review and Comparison of Certain Methods of Computer Performance Evaluation
J Meredith Smith · ITNOW · 1968
In this paper we review existing methods for evaluating the performance of various aspects of computer systems and discuss the advantages and limitations of methods commonly used. There are essentially two approaches to evaluating the performance of a system. We either set the system various tasks (benchmarks) and determine how well it performs them, or we analyse the system in terms of specific parameters and derive a measure of performance. The first method assesses the system as a whole but the benchmark chosen may not accurately represent the nature of the work to be performed. Combining the results of several benchmarks partially alleviates these criticisms, but choosing the right combination of benchmarks is difficult. To assess a system analytically we have to consider individual parameters and determine how they should be combined. Comparisons have often been based, for instance, on core cycle times and on instruction mixes, ie the time taken to perform a particular frequency distribution of instructions. The suggestion has been made by Knight [5] that the store size can be allowed for by multiplying the reciprocal of the mix value by the size in bits. A multiplicative factor due to the input/output transfer rate and the number of channels has sometimes been suggested. The question arises, however, as to whether these factors are both linear and multiplicative. Performance is affected by a variety of factors.