The case for application-specific benchmarking
Margo Seltzer, D. Krinsky, Keith Smith, Xiaolan Zhang · 2003
Most performance analysis today uses either microbenchmarks or standard macrobenchmarks (e.g. SPEC, LADDIS, the Andrew benchmark). However, the results of such benchmarks provide little information to indicate how well a particular system will handle a particular application. Such results are, at best, useless and, at worst, misleading. In this paper we argue for an application-directed approach to benchmarking, using performance metrics that reflect the expected behavior of a particular application across a range of hardware or software platforms. We present three different approaches to application-specific measurement, one using vectors that characterize both the underlying system and an application, one using trace-driven techniques, and a hybrid approach. We argue that such techniques should become the new standard.