Error Estimation and Model Selection
Tobias Scheffer · 1999
In order to select a good hypothesis language (or model class) from a collection of possible model classes, we have to assess the generalization performance of the hypothesis which is returned by a learner that is bound to use a particular model class. This abstract of my doctoral dissertation deals with an analysis of the expected error rate of classifiers that leads to a new and very efficient way of assessing this error rate. Similar analyses can be applied to quantify the generalization performance of a holdout testing based model selection algorithm, and to quantify the optimistic bias of the error estimate which is imposed by running several learners on the same data set and selecting the one with the lowest holdout error rate. The analysis provides a model selection algorithm which can solve model selection (e.g., feature subset selection) problems with as many as 10,000 attributes and 12,000 examples. 1 Introduction In the setting of classification learning, the task of a lear...