Arguing Practical Significance in Software Engineering Using Bayesian Data Analysis.
Richard Torkar, ROBERT H. FELDT, Carlo A. Furia · arXiv (Cornell University) · 2018
This paper provides a case for using Bayesian data analysis (BDA) to make more informed claims regarding practical significance of software engineering research. We show that using BDA, here combined with cumulative prospect theory (CPT), is appropriate when a researcher or practitioner wants to make more explicit connections between statistical findings and practical significance in empirical software engineering research. To illustrate our point, we provide an example case using previously published data. We build a multilevel Bayesian model for this data, for which we compare the out of sample predictive power. Finally, we use our model to make out of sample predictions while, ultimately, connecting this to practical significance using CPT. Throughout the case that we present, we argue that a Bayesian approach is a natural, theoretically well-grounded, practical work-flow for data analysis in empirical software engineering. By including prior beliefs, assuming parameters are drawn from a probability distribution, assuming the true value is a random variable for uncertainty intervals, using counter-factual plots for sanity checks, conducting predictive posterior checks, and out of sample predictions, we will better understand the phenomenon being studied, while at the same time avoid the obsession with $p$-values.