Investigating benchmark correlations when comparing algorithms with parameter tuning
Lee A. Christie, Alexander E. I. Brownlee, John Robert Woodward · Proceedings of the Genetic and Evolutionary Computation Conference Companion · 2018
Benchmarks are important for comparing performance of optimisation algorithms, but we can select instances that present our algorithm favourably, and dismiss those on which our algorithm under-performs. Also related are automated design of algorithms, which use problem instances (benchmarks) to train an algorithm: careful choice of instances is needed for the algorithm to generalise.