Significance testing with incompletely randomised cases cannot possibly work.
Stephen A.C. Gorard · Durham Research Online (Durham University) · 2018
This brief paper illustrates why the use of significance testing cannot possibly work with incompletely randomised cases. The first section reminds readers of the logical argument of “denying the consequence”, and the fallacy of trying to affirm the consequence, of a set of premises. The second section extends the argument of the denying the consequence to the weaker situation where there is uncertainty, and the third shows that this weaker situation is the „logical‟ basis for the practice of significance testing when analysing data. The fourth section looks at how the same argument becomes a fallacy when conducting significance tests with incompletely randomised or non-random cases. The final section summarises the implications for analysts, and for their future analyses and reporting.