Betting After the Race is Over: The Perils of Post Hoc Hypothesizing
James Lytton Payne, James A. Dyer · American Journal of Political Science · 1975
Using a recent article by Sidney Ulmer as an example, this note offers two bits of advice to statistical researchers. One must take care to specify his hypotheses on an a priori basis. One must be cautious when the number of observations approaches the number of independent variables. In the popular movie The Sting, a variation on an old con game is depicted. The con men fleece the pigeon by inducing him to bet on a horse race which, unbeknownst to the pigeon, has already been run. As obvious a ploy as this is, we often fall victim to the same con when we accept statements about the significance of relationships that were discovered after the data were analyzed. Whenever we find a relationship between variables our first question should be: Would this relationship have occurred by chance? That is, if we had started with random numbers (conforming to the same distribution as our data), would we have done as well? If random numbers tend to give correlations as strong as our research result, then we cannot say we have a finding. The purpose of tests of statistical significance is to provide an estimate of the results random values would yield for the statistic in question. Thus, the researcher is spared the task of computing the statistic many times with different sets of random numbers to determine how frequently random numbers yield a result as high as that obtained with his data. Tests of significance, however, will serve their purpose only if the assumptions they are based on are met. Perhaps the most important assumption made by virtually all tests of significance is that the relationship in question has been specified a priori. The problem is that many research designs rely, to varying degrees, on the post hoc selection of hypotheses. Computers make the calculation of relationships so easy that a researcher has little incentive to limit his analysis to a