The Problems with “The Problem with ‘Magnitude-based Inference’”
Alan M. Batterham, Will G. Hopkins · Medicine & Science in Sports & Exercise · 2018
Dear Editor-in-Chief, Kristin Sainani offered numerous reasons why magnitude-based inference (MBI) should not be used (1). We have provided a detailed rebuttal (sportsci.org/2018/mbivind.htm), which we summarize here. She claims that the probabilistic statements in MBI, such as the treatment is possibly beneficial, are invalid, because these are Bayesian statements and MBI is not Bayesian. We assert that this claim is false because MBI is a legitimate form of reference Bayesian inference with a dispersed uniform prior (e.g., (2)); therefore, the probabilities provided by MBI are objective trustworthy estimates of uncertainty in the true value. Sainani supports instead “qualitative judgments” of the lower and upper confidence limits, without realizing that the level of confidence renders such judgments quantitative, and they are essentially MBI and Bayesian. Although Bayesians seldom concern themselves with error rates, there is an ethical imperative to assist clinicians and practitioners with decisions, which have attendant errors. Sainani regards as “specious” the logic in nonclinical MBI that there is no type I error when the true effect is trivial and the MBI outcome is likely substantial because the effect is also unlikely trivial (e.g., with a probability of 0.06). However, according to her logic, “specious” would also apply to failure to declare a type I error in null hypothesis significance testing (NHST), when the true effect is zero and the outcome is nonsignificant (e.g., with a P value of 0.06). She shows that our definitions of error “wildly underestimate” type II error rates, but her estimates are based on the null hypothesis, which some eminent statisticians now regard as an untrustworthy approach to inference (e.g., (3); see the supplement). She highlights the high type I error rates for clinical MBI, yet these are comparable with those of NHST over the range of small sample sizes (suboptimal for NHST) and trivial effect magnitudes; where MBI type I rates are high, they occur mostly with effects presented to the clinician or practitioner as only possibly beneficial, whereas those for NHST are presented as definite “real” effects. She claims that unclear outcomes in MBI (inadequate precision in estimates of effects) should be counted as inferential errors. We reject this claim on the grounds that an error does not occur until a decision is made about the true magnitude. We previously adopted this reasoning even-handedly with conservative NHST and showed that error rates, rates of decisive outcomes, and publication bias were generally superior with MBI (4). The smaller sample sizes for publishability with MBI also entail less risk of unethically underpowered studies. She claims that the publications we cited (5–7) as evidence supporting the theoretical basis of MBI “do not provide such evidence.” We contend that careful reading of these references and our full rebuttal shows this claim to be false. In conclusion, her recommendation that MBI should not be used is itself based on unsound or demonstrably false assertions. We reassure researchers that MBI represents a valuable advance on NHST, with the benefits of Bayesian probabilistic inference and without the drawback of a subjective prior. MBI should be used.