Tutorial on Model Comparison Methods
Jay I. Myung, Mark A. Pitt · eScholarship (California Digital Library) · 2011
Tutorial on Model Comparison Methods Jay I. Myung ([email protected]) Department of Psychology, Ohio State University 1835 Neil Avenue, Columbus, OH 43210 USA Mark A. Pitt ([email protected]) Department of Psychology, Ohio State University 1835 Neil Avenue, Columbus, OH 43210 USA Keywords: Cognitive modeling; model evaluation; model selection; model complexity; Occam’s razor; Akaike Information Criterion; cross-validation, minimum description length; Bayes factor; NHST-based likelihood ratio test. enough to capture the bowed shape of the data. The model in the middle graph is a bit more complex and does a good job of fitting only the regularity in the data. Because of this, the goodness of fit and generalizability measures are higher and also similar. Where the two functions diverge is when the model is more complex than is necessary to capture the main trend. The model in the right-most graph captures the experiment-specific noise, fitting every data point perfectly. Goodness of fit rewards this behavior by yielding an even higher fit score, whereas generalizability does just the opposite, penalizing the model for its excess complexity. Background The question of how one should choose among competing explanations of data is at the heart of the scientific enterprise. Computational models of cognition are increasingly being advanced as descriptions and explanations of behavior in the cognitive sciences. The success of this line inquiry depends on the availability of robust quantitative methods to guide the evaluation and selection of these models. The basic tenet in model comparison may be summarized in terms of the following three quantitative criteria of model evaluation: Goodness of fit Overfitting Goodness of fit: A good fit (or simulation of observed data) is a necessary, but not a sufficient, condition for judging the adequacy of a model; Generalizability Complexity: When comparing models, one should avoid choosing an unnecessarily complex model that overfits, and instead, should try to identify a model that is sufficiently complex, but not too complex, to capture the regularity in the data; Model Complexity Generalizability: Model comparison should be based not upon how well a model fits a particular pattern of observed data, but upon generalizability, which refers to how well a model fits not only the observed data at hand but also new, as yet unseen, data samples from the same underlying process that generated the observed data. Figure 1. The relationship among goodness of fit, complexity, and generalizability. Reprint from Pitt and Myung (2002). The foremost goal of model comparison is achieving good generalizability by trading off between two opposing forces, goodness of fit and complexity. Scientists in statistics and computing sciences have proposed a number of model comparison methods that formally implement this trade-off principle. The purpose of this tutorial is to introduce these methods of model comparison to cognitive scientists who are engaged in computational modeling of cognitive behavior. Our goal in the tutorial is to provide a good conceptual overview of the methods with illustrative examples using selected models in cognitive science. By the end of the tutorial, the attendee should have a grasp of the basic issues The relationship among the three criteria is illustrated in Figure 1. In the figure, goodness of fit and generalizability are represented as curves whose performance can be compared as a function of complexity. The three smaller graphs in the lower panel contain the same data set (dots) and the fits to these data by increasingly more complex models (lines). The left-most model in the figure underfits the data. The data are curvilinear whereas the model is linear. In this case, goodness of fit and generalizability produce similar outcomes because the model is not complex