Rethinking Replication in Software Engineering: Can We See the Forest for the Trees?
Mika Mäntylä, Casper Lassenius, Jari Vanhanen · 2010
In this paper, we argue that the concept of replication of empirical studies in software engineering should be understood more broadly than it currently is. In particular, the replication of case studies and surveys as a way of validating and extending theories should be incorporated in the mainstream view of replication, which at present is mostly focused on controlled experiments. A small-sample study of papers in IEEE Transactions on Software Engineering shows that about 10 % of studies published in 2009 can be considered replications. However, none of these was self-labeled as replication. We think that the authors believed that labeling their work as replication might decrease its value in the eyes of reviewers and editors. We conclude that there is no acute shortage of replication studies in software engineering if taking a broader viewpoint to replication, but the definition and valuation of ‘replication studies ’ need to be re-evaluated in the software engineering community.