Analyzing the impact of undersampling on the benchmarking and configuration of evolutionary algorithms
Diederick Vermetten, Hao Wang, Manuel López‐Ibáñez, Carola Doerr, Thomas Bäck · Proceedings of the Genetic and Evolutionary Computation Conference · 2022
The stochastic nature of iterative optimization heuristics leads to inherently noisy performance measurements. Since these measurements are often gathered once and then used repeatedly, the number of collected samples will have a significant impact on the reliability of algorithm comparisons. We show that care should be taken when making decisions based on limited data. Particularly, we show that the number of runs used in many benchmarking studies, e.g., the default value of 15 suggested by the COCO environment, can be insufficient to reliably rank algorithms on well-known numerical optimization benchmarks.