Editorial: Testing for real!
Martin Woodward · Software Testing Verification and Reliability · 2006
Testing for real!Academics are often chided for doing research that is only of theoretical interest and has no practical significance.Many outside the academic world would probably argue that a dose of reality would do us good and that, to use common parlance, academics should 'get real'.The theme of software testing in the real world was highlighted in the opening keynote talk at UK-Test 2005, the Third U.K. Workshop on Software Testing Research held at the University of Sheffield in the U.K. in September 2005 * .Paul Gibson, the Development Operations Manager at IBM Hursley, spoke on 'Testing Challenges for IBM'.Having reminded the audience of the range of IBM software in the marketplace, Gibson started by quoting Watts Humphrey who, in the past, has stated that 'coders introduce bugs at the rate of 4.2 defects per hour of programming' † .A more recent statement by Humphrey [1] puts the figure at the higher rate of 4.6 defects per hour for experienced developers designing as they code, with a lower rate of 2.0 defects per hour for developing a detailed design in advance of coding.Anyway, to emphasize his case, Gibson extrapolated the figure for defects per hour to around 400 per month and about 5000 per year, a figure not untypical apparently for the total number of defects in an IBM project.As Gibson stressed, one of the main challenges for IBM is to remove defects early, a familiar message, but one that was dramatically reinforced by considering the potential costs of finding defects ever later in the development process, with the worst scenario being faults found in operational software.The CICS system was mentioned: this transaction processing system is used by numerous banks and financial organizations world-wide and the economic impact if services are interrupted just for a short period is staggering.Even for something less critical, such as an e-commerce application, the cost of customer down time was estimated at U.S. $480 000 per hour.IBM use a diverse set of validation and testing strategies, but perhaps one technique that deserves greater awareness (at least, if the rather disappointing show of hands is any guide, by those in the audience at UK-Test 2005 who were familiar with it) is orthogonal defect classification (ODC).This idea, introduced by Chillarege and colleagues [2,3], aims to categorize the space of all defects into a small number of broad orthogonal categories.Repeated application of ODC has shown that different process stages have different signature patterns of defect distribution; for example, one might expect more defects of 'interface' type to be found during integration testing.Hence, measuring the numbers of defects of each type helps identify progress and can also drive appropriate corrective action.It is particularly pleasing that both papers in this issue of STVR are especially relevant and practical for the software industry.Belli, Budnik and White, in their paper, concentrate on testing graphical user interfaces (GUIs).Building on earlier work, they provide a coherent strategy for testing GUIs * See http://www.uktest.