Is the perfect the enemy of the good
Sean Luke, Liviu Panait · 2002
Much of the genetic programming literature compares techniques using counts of ideal solutions found. These counts in turn form common comparison measures such as Koza’s Computational Effort or Cumulative Probability of Success. The use of these measures continues despite past warnings that they are not statistically valid. In this paper we too criticize the measures for serious statistical problems, and also argue that their motivational justification is faulty. We then present evidence suggesting that idealsolution counts are not necessarily positively related to best-fitness-of-run statistics: in fact they are often inversely correlated. Thus claims based on ideal-solution counts can mislead readers into thinking techniques will provide superior final results, when in fact the opposite is true. 1