Survey-Based Insights into the Replication Crisis and the 3R in Software Engineering
Ivanildo Monteiro de Azevedo, Ana Paula Vasconcelos, Eudis Teixeira, Sérgio Soares · 2025
Context: Efforts to improve reproducibility and research credibility have gained relevance in multiple fields, including Software Engineering (SE), where the 3R practices (Repeatability, Reproducibility, and Replicability) are essential to ensure the reliability of empirical studies. Despite growing interest in Open Science, concerns about a Replication Crisis persist. Objectives: To assess the perceptions of SE researchers of the Replication Crisis and 3R practices, identify good practices, barriers, and facilitators to reproducible research, and evaluate the community’s acceptance of the ACM’s standardized definitions of 3R. Method: We conducted a survey adapted from Baker [5], targeting authors of SE studies related to replication. From a list of 1,061 researchers, we received 101 responses. The questionnaire combined Likert-scale and open-ended questions. Responses were analyzed using descriptive statistics and Reflexive Thematic Analysis. Results: Most respondents acknowledged the importance of 3R practices. On average 84.5% agreed with the ACM definitions, participants raised concerns about their clarity and applicability, especially to qualitative research. 74.3% recognized the existence of a Replication Crisis in SE. The key challenges reported include a lack of protocols, selective reporting, data unavailability, and pressure to publish. Positive actions included using containers, version control, artifact sharing, and Open Science practices. However, participants noted that cultural and institutional incentives for reproducibility are limited. Conclusion: Although SE researchers support the principles of 3R practices and recognize ongoing challenges, uncertainty persists about the scope and solutions of the crisis. This study highlights the need for more precise terminology, better reporting standards, and greater institutional support to promote reproducibility, transparency, and research integrity in SE.