More on finding a single number to indicate overall performance of a benchmark suite
Lizy K. John · ACM SIGARCH Computer Architecture News · 2004
The topic of finding a single number to summarize overall performance over a benchmark suite is continuing to be a difficult issue 14 years after Smith’s paper [1]. While significant insight into the problem has been provided by Smith [1], Hennessey and Patterson [2], Cragon [3], etc, the research community still seems to be unclear on the correct mean to use for different performance metrics. How should metrics obtained from individual benchmarks be aggregated to present a summary of the performance over the entire suite? What are valid central tendency measures over the whole benchmark suite for speedup, CPI, IPC, MIPS, MFLOPS, cache miss rates, cache hit rates, branch misprediction rates, etc? Arithmetic mean has been touted to be appropriate for