An Investigation of p-Hacking in E-Commerce A/B Testing
Alex P. Miller, Kartik Hosanagar · Information Systems Research · 2025
Concerns about the integrity of statistical analyses have risen in recent years, with particular attention given to “p-hacking”—a process whereby analysts conduct several statistical tests until they achieve a statistically significant result. Although extensively studied in academic settings, less is known about its prevalence in industrial contexts. In this study, we investigate whether p-hacking occurs in e-commerce A/B testing. We analyzed nearly 2,300 A/B tests conducted by hundreds of firms using a large A/B testing platform. Such platforms typically offer continuous monitoring of test results, a feature that facilitates real-time decision making but also enables potential p-hacking through selective stopping or continuation of experiments. Contrary to concerns raised by earlier research on academic practices, we found no significant evidence of p-hacking in our sample. These findings suggest that the industrial application of experimentation may be less susceptible to p-hacking than academic research. We discuss several possible factors explaining the divergent results, highlighting the potential role of organizational learning and the importance of economic incentives. Our study contributes to the broader discussion on research integrity and underscores the importance of considering contextual factors in assessing statistical malpractice.