Illustrating the trade-off between time, quality, and success probability in heuristic search
Ivan Ignashov, Буздалова Арина Сергеевна, Maxim Buzdalov, Carola Doerr · Proceedings of the Genetic and Evolutionary Computation Conference Companion · 2019
Benchmarking aims to investigate the performance of one or several algorithms for a set of reference problems by empirical means. An important motivation for benchmarking is the generation of insight that can be leveraged for designing more efficient solvers, for selecting a best algorithm, and/or for choosing a suitable instantiation of a parametrized algorithm. An important component of benchmarking is its design of experiment (DoE), which comprises the selection of the problems, the algorithms, the computational budget, etc., but also the performance indicators by which the data is evaluated. The DoE very strongly depends on the question that the user aims to answer. Flexible benchmarking environments that can easily adopt to users' needs are therefore in high demand.