Using metalearning to predict when parameter optimization is likely to improve classification accuracy
Parker Ridd, Christophe G. Giraud-Carrier · 2014
Abstract. Work on metalearning for algorithm selection has often been criticized because it mostly considers only the default param-eter settings of the candidate base learning algorithms. Many have indeed argued that the choice of parameter values can have a signif-icant impact on accuracy. Yet little empirical evidence exists to pro-vide definitive support for that argument. Recent experiments do sug-gest that parameter optimization may indeed have an impact. How-ever, the distribution of performance differences has a long tail, sug-gesting that in most cases parameter optimization has little effect on accuracy. In this paper, we revisit some of these results and use met-alearning to characterize the situations when parameter optimization is likely to cause a significant increase in accuracy. In so doing, we show that 1) a relatively simple and efficient landmarker carries sig-nificant predictive power, and 2) metalearning for algorithm selection should be effected in two phases, the first in which one determines whether parameter optimization is likely to increase accuracy, and the second in which algorithm selection actually takes place. 1