Model Complexity, Goodness of Fit and Diminishing Returns
Igor V. Cadez, Padhraic Smyth · 2000
We investigate a general characteristic of the trade-o in learning problems between goodness-of-t and model complexity. Speci- cally we characterize a general class of learning problems where the goodness-of-t function can be shown to be convex within rstorder as a function of model complexity. This general property of \\diminishing returns" is illustrated on a number of real data sets and learning problems, including nite mixture modeling and multivariate linear regression. 1 Introduction, Motivation, and Related Work Assume we have a data set D = fx 1 ; x 2 ; : : : ; xn g, where the x i could be vectors, sequences, etc. We consider modeling the data set D using models indexed by a complexity index k, 1 k kmax . For example, the models could be nite mixture probability density functions (PDFs) for vector x i 's where model complexity is indexed by the number of components k in the mixture. Alternatively, the modeling task could be to t a conditional regression mode...