Some theoretical comparisons of stochastic optimization approaches
James C. Spall, Stacy D. Hill, Don Stark · 2000
This paper is a first step to formal comparisons of several leading optimization algorithms, establishing guidance to practitioners for when to use or not use a particular method. It focuses on four general algorithm forms: random search, simultaneous perturbation stochastic approximation simulated annealing, and evolutionary computation. We summarize the available theoretical results on rates of convergence for the four algorithm forms and then use the theoretical results to draw some preliminary conclusions on the relative efficiency. Our aim is to sort out some of the competing claims of efficiency and suggest a structure for comparison that is more general and transferable than the usual problem-specific numerical studies. Much work remains to be done to generalize and extend the results to problems and algorithms of the type frequently seen in practice.