10. Benchmarks
Society for Industrial and Applied Mathematics eBooks · 2001
In this chapter a sketch of the benchmarking arena is attempted and some widely utilized benchmarks will be presented. Given the extent of this activity, at both a scientific and a technical level, but also related to computer procurement, we cannot render justice to it in a short section. We refer the reader to comprehensive books on performance analysis and benchmarking such as [11] and [6]. Benchmarks can be classified according to the complexity of the measurement and of the code on which they are based. Low-level benchmarks aim at providing data for simple, low-level, architectural parameters such as the latency and the bandwidth of the network. “Parkbench” (http://www.netlib.org/parkbench/), for example, contains a collection of such benchmarks. Moving upwards in the hierarchy, we distinguish kernel benchmarks, measuring performance of certain loop constructs or compiler optimizations of simple code fragments. The “Liver-more loops” (http://www.llnl.gov/asci_benchmarks/asci/limited/lfk/asci_lfk.html) are representative of such benchmarks. Higher up in the hierarchy are benchmarks for certain well-defined mathematical algorithms such as the solution of linear systems, simple multigrid schemes, etc. LINPACK and the NAS parallel benchmarks are important examples in this category. At the highest level, benchmarks for entire applications have been proposed. However, due to the complexity of the analysis and optimization and the high number of degrees of freedom in interpreting benchmarks at this level, full application benchmarks are not nearly as widely utilized as the others. Benchmarks are prescribed in various forms. Some of the higher level benchmarks are aimed at assessing the best level of performance that can be achieved on a machine. They specify only the mathematical problem that needs to be solved and leave complete freedom to all the implementation details. The NAS parallel benchmarks (http://www.nas.nasa.gov/Software/NPB) are the most representative suite of pencil-and-paper defined benchmarks. Other benchmarks provide source code that must not be modified, relying entirely on the compiler for optimizations. One of the best known of the floating point benchmarks is the LINPACK benchmark (http://www.netlib/org/). One class of this benchmark is aimed at single processors and SMPs, and another class at massively parallel machines.