Debugging observational data studies
Anna Dreber, Magnus Johannesson · 2025
In this chapter, we discuss various approaches for increasing the credibility of studies based on observational data. Most observational data studies are based on retrospective data where it is difficult to use preregistrations with detailed pre-analysis plans to improve credibility. This is both because the researchers may already have access to the data so that preregistrations are not credible and because it is challenging to foresee all the necessary analytical decisions in complex datasets prior to looking at the data. Alternative procedures may therefore be needed to improve credibility, and we discuss different possible approaches divided into if the observational data is already in the public domain or not. We end the chapter with some lessons and recommendations.