Model selection using measure functions
Arne Andersson, Paul Davidsson · 1998
Introduction In this work, we suggest a new approach to model selection and evaluation. Today, most methods for evaluating the quality of a learned model (classifier) is based on some kind of cross-validation [6]. However, we argue that it is possible to make evaluations that take into account other important aspects of the model than just classification accuracy on a few instances. Our approach is based on measuring explicit properties of the learned model rather than properties of the algorithm that produced the model. Therefore, in contrast to for example Nakhaeizadeh and Schnabl [9], we pay no attention to properties such as the employed algorithm's time and space complexity. For each possible combination of a training set and a classifier, a measure function assigns a value describing how good the classifier is. Measure functions have a number of favorable properties from both a theoretical and a practical point of view. They provide a complemen