The Chrysalis Effect: How ugly data metamorphosize into beautiful articles
Ernest H. O’Boyle, George C. Banks, Erik N Gonzalez-Mule · Academy of Management Proceedings · 2013
The issue of a published literature not representative of the population of research is most often discussed in terms of entire studies being suppressed. However, alternative sources of publication bias are questionable research practices (QRPs) that entail post hoc alterations of hypotheses to support data or post hoc alterations of data to support hypotheses. Using General Strain Theory as an explanatory framework, we outline the means, motives, and opportunities to engage in QRPs. We then assess the frequency of these QRPs by identifying and tracking differences of dissertations that were subsequently published in refereed journals. Our primary finding is that from dissertation to journal publication, the ratio of supported to unsupported hypotheses more than doubles (.85 to 1.00 versus 2.00 to 1.00). This rise in predictive accuracy was directly attributable to the dropping of non-significant hypotheses, the addition of statistically significant hypotheses, the reversing of the predicted direction of hypotheses, and data manipulation. We conclude with a research agenda to help mitigate the problem of an unrepresentative literature that we label, the Chrysalis Effect.