Benchmarks and watermarks
Robert G. Estell · ACM SIGMETRICS Performance Evaluation Review · 1980
Historically, benchmark tests have been one of several ways to size a computer system, and measure its performance. Today, it is more important to test the System Software than the machine hardware. (Thus the term "watermark" (as on bond paper) for software tests.) Watermarks of software suffer the same limitations and risks as benchmarks of hardware: e.a., they should be supplemented with simulations, models, and other analysis and design tools of our trade. Perhaps most significantly, watermarks, like benchmarks, can be biased by their creators.