Benchmarking autonomic capabilities: promises and pitfalls
Aaron B. Brown, Joseph M. Hellerstein, Matt Hogstrom, Tzen Yik Lau, Sam S. Lightstone, Peter Shum, Matt A. Yost · 2004
Benchmarks provide a way to quantify progress in a field. Our goal is to produce a suite of benchmarks covering the four categories of autonomic capabilities: self-configuring, self-healing, self-optimizing, and self-protecting (IBM, 2003). This is not an easy task, however, and in this paper we identify several of the challenges and pitfalls that must be confronted to extend benchmarking technology beyond its traditional basis in performance evaluation.