Evaluating software development by analysis of change data
David M. Weiss · 1981
An effective data collection method for evaluating software development methodologies and for studying the software development process is described. The method uses goal-directed data collection to evaluate methodologies with respect to the claims made for them. The data to be collected is based on the changes made to the software during development. To insure accuracy of the data, validation is performed concurrently with software development and data collection. Validation is based on interviews with those people supplying the data. Feasibility of the data collection method is demonstrated by its application to five different projects in two different environments. (NASA's Goddard Space Flight Center (GSFC) and the Naval Research Laboratory (NRL). All of the NASA projects were monitored as part of NASA's Software Engineering Laboratory). Results concerning the data collection methodology show that the methodology is feasible and useful, and that data validation is a necessary part of change data collection. Without it, as much as 50% of the data may be erroneous. Results concerning the projects show that the NRL methodology is effective in meeting several of its claims, especially those concerning ease of change of the resulting software. The NASA and NRL software showed several similar characteristics. As an example, for all projects, more than half of all changes took an hour or less to design. In both environments most errors involved the design and implementation of individual programs, rather than interfaces among programs. None of the projects studied showed any tendency towards a ripple effect resulting from changes. Other results include distributions showing sources of change, sources of errors, difficulty of finding causes of errors, frequency distributions of changes according to number of components changed, changes according to subsystem, difficulty of change according to source of change, and source of error according to programmer.