Exploring Data Cleanness in Defects4J and Its Influence on Fault Localization Efficiency
Md Nakhla Rafi, An Ran Chen, Tse-Hsun Peter Chen, Shaohua Wang · 2024
Defects4J stands out as the most popular benchmark in software testing research, known for its comprehensive collection of real bugs from open-source systems. This paper presents an in-depth study of Defects4J's fault-triggering tests, particularly examining the influence of developer modifications post-bug reports on spectrum-based fault localization (SBFL) techniques. Our findings reveal that 55% of these tests were newly added and 22% modified with developer knowledge, impacting the accuracy of SBFL. Notably, SBFL techniques' performance drops significantly (up to -415% in Mean First Rank) when developer knowledge is absent in the tests. We provide a curated dataset of bugs without this knowledge, facilitating more realistic evaluations of SBFL techniques using Defects4J. This research offers insights for the development of future bug benchmarks.