Measurement-Driven Prediction of High Error and High Effort Software Components in Large-Scale Systems
Richard W. Selby · Space 2004 Conference and Exhibit · 2004
This paper illustrates two synergistic strategies for enabling lar ge -scale software development and management. Current and future technologic systems are increasingly software -intensive and large -scale in terms of system size, component breadth and maturity, and development heterogeneity. Developers who tackle these l arge -scale systems and attempt to achieve ambitious goals often produce unfavorable design flexibility, progress visibility, and schedule duration. We summarize two interrelated approaches for developing these types of systems. First, we describe the Ama deus measurement -driven analysis and feedback system that provides features for automating measurement of software processes and products. Amadeus provides a set of capabilities for enabling empirically guided software development of large systems, includ ing capabilities for specifying empirical analyses, collecting the underlying data, and feeding the results back into development processes. The system increases technical and management visibility into large systems, can automate measurements based on re gular product synchronizations, and has been used at many organizations. The Amadeus system embodies architectural principles and abstract interfaces for measurement -driven analysis and feedback systems, and it serves as an extensible integration framewor k for empirically based analysis techniques. Second, we describe metric -driven decision tree and decision network models that classify software components in large systems according to their likelihood of having user -specified properties such as high erro r-proneness or high development effort. The metric -driven decision models enable coarse -grain analysis of large -scale, multi -component heterogeneous systems and can direct the application of fine -grain analysis techniques to identify high -payoff areas for error detection or redesign. The decision models serve as metric integration mechanisms that enable the synergistic use of numerous metrics simultaneously and can integrate measurements collected by Amadeus or other systems. Analysis techniques automati cally generate the decision models to calibrate them to new projects and organizations. We evaluate the predictive effectiveness of the decision models in terms of consistency and completeness using error and effort data from large NASA systems. Consiste ncy is defined as 100% minus the percent of false positives, and completeness is defined as 100% minus the percent of false negatives. On average, the decision network models achieved 79% consistency and 69% completeness in predictions of high error and h igh effort software components. We describe the prediction results and outline several customizations that enable tradeoffs for optimizing different prediction criteria. In conclusion, we outline future research directions that build on these strategies and ideas.