A statistical evaluation of the impact of parameter selection on storage system benchmarks
Nohhyun Park, Weijun Xiao, Kyubaik Choi, David J. Lilja · 2011
The performance evaluation of storage systems is a difficult task due to lack of representative workloads. Storage benchmark tools test a single aspect such as random read performance at a time. While this approach does allow us to gain useful insights to the system behavior, current practices are largely inadequate due to large benchmark input parameter space. For example, a benchmark with 10 parameters require at least 1024 experiments if conducted exhaustively even if we only test extreme values of each parameter. Furthermore, there is no way to tell if the benchmark being used is enough to test all realistic scenarios. As a result, researchers and developers rely on multiple benchmarks with ad hoc input parameters. We propose a method to quickly identify input parameters that have high effect on the performance metric of interest. We also show that using multiple benchmarks is unnecessary at times and a good benchmark can cover all operational space by providing a control over key parameters that affect the performance metric being measured. 1 introduction Storage system performance is one of the most critical factors in meeting overall system performance expectations. The performance variances of typical storage systems today can vary by several orders of magnitude [1]. It is very easy to get misled by performance numbers without understanding the relationship between the storage workload and its performance variances. Characterizing the workload for storage systems is typically much more complex than it is for processor. In a sim-