On Overfitting Avoidance as Bias
David H. Wolpert · 1993
: In supervised learning it is commonly believed that Occam's razor works, i.e., that penalizing complex functions helps one avoid "overfitting" functions to data, and therefore improves generalization. It is also commonly believed that cross-validation is an effective way to choose amongst algorithms for fitting functions to data. In a recent paper, Schaffer (1993) presents experimental evidence disputing these claims. The current paper consists of a formal analysis of these contentions of Schaffer's. It proves that his contentions are valid, although some of his experiments must be interpreted with caution. In doing so, it proves that there are "as many" scenarios in which a learning algorithm using cross-validation fails as in which it succeeds (and similarly for any other learning algorithm), as far as off-training set behavior is concerned. Interestingly, this proof also indicates that there are as many scenarios in which use of a test set fails to accurately predict behavior off ...