Ready to ROC? A Tutorial on Simulation-Based Power Analyses for Null Hypothesis Significance, Minimum-Effect, and Equivalence Testing for ROC Curve Analyses
Paul Riesthuis, Henry Otgaar, Charlotte A. Bücken · 2024
The receiver operating characteristic (ROC) curve and its corresponding (partial) area under the curve ([p-]AUC) are frequently used statistical tools in psychological research to assess the discriminability of a test, method, intervention, or procedure. In this paper, we provide a tutorial on how to conduct simulation-based power analyses for ROC curve and (p-)AUC analyses in R. We also created a ShinyApp to perform such power analyses. In our tutorial, we highlight the importance of setting a smallest effect size of interest (SESOI) for which researchers want to conduct their power analysis. The SESOI is the smallest effect that is practically or theoretically relevant for a specific field of research or study. We provide how such a SESOI can be established and how it changes hypotheses from simply establishing whether there is a statistically significant effect (i.e., null-hypothesis significance testing) to whether the effects are practically or theoretically important (i.e., minimum-effect testing) or whether the effect is too small to care about (i.e., equivalence testing). We show how power analyses for these different hypothesis tests can be conducted via a confidence interval focused approach. This confidence interval focused simulation-based power analysis can be adapted to different research designs and questions and improves the reproducibility of power analyses.