Cross- vs Within-Company Defect Prediction Studies
Tim Menzies, Burak Turhan, Ayşe Bener, Justin Distefano · 2007
In a recent May 2007 IEEE TSE article, Kitchenham et.al. explored effort estimation and found contradictory evidence about the value of cross- vs within-company data. Those contradictory results may have been the result of effort estimation features, some of which are subjective in nature. Static code features are different than effort estimation features. They can be generated in an automatic, rapid, and uniform manner across multiple projects. Therefore, in theory, the conclusions reached from such features may be more uniform. This paper tests that theory by searching for uniform conclusions using cross- or within-company static code features. Whereas Kitchenham et.al. explored effort estimation, this paper explores defect prediction. Cross-company static code features will be found to generate higher false alarm rates than within-company data. Hence, cross-company data is best used for mission critical software where (a) the extra costs associated with high false alarm rates is compensated by (b) an associated increase in the probability of predicting fault modules. For other classes of software, false alarm rates can be decreased using a very small amount of local data (often, just 100 modules). In our experiments, the use of within-company data halved the false alarm rate while decreasing prediction rates by only ≈ 10%. Hence, for non-mission-critical software, we strongly recommend using within-company data for defect prediction.