Editorial: The merits of diversity
Martin R. Woodward · Software Testing Verification and Reliability · 2003
The merits of diversityAs well as our regular book reviews and diary, this issue of STVR contains three papers on a diverse range of topics: regression testing, reliability modelling and testing UML designs.This variety is welcome: it adds interest and is likely to engage more readers.The idea that there is merit in diversity is a notion that has been much in my mind just recently, as a result of involvement in a project to assess the evidence comparing fault elimination strategies for software.One of the main conclusions was that many techniques appear to be complementary and the best advice when high levels of assurance are required is to use a battery of different approaches.Faults that one particular technique fails to expose will hopefully be exposed by some other technique.Assessing the relative worth of the many various testing and analysis techniques remains one of the great challenges for our community.Indeed, it may never be possible to characterize fully the efficacy of each individual technique.Using a battery of techniques is not the only way in which diversity helps with fault exposure.Testing professionals are well aware that it makes sense to develop test cases from a variety of artifacts: requirements, specification, design and code.The shortcomings of test data generated from one source may be made good by test data generated from another source.The coverage of potential failureinducing domains will surely be improved by the diverse origins of the data.Of course the notion of diversity appears even more directly in multiversion software, the intuition being that independently developed programs in conjunction with majority voting enhance reliability and result in greater fault tolerance.As it turns out, there is experimental evidence to suggest that coincident failures reduce the effectiveness of majority voting in handling faults [1], so the extra cost of producing multiple versions should not automatically lead to a reduction in testing techniques used.Once again diversity should be maintained.Turning to the contents of this issue, the first paper, by Sebastian Elbaum, Praveen Kallakuri, Alexey Malishevsky, Gregg Rothermel and Satya Kanduri, is entitled 'Understanding the effects of changes on the cost-effectiveness of regression testing techniques'.Regression testing is the process of testing modified software; determining which of the test cases for the original program to reuse and what new test cases to add, can lead to diverse strategies.In addition, a variety of techniques exist that prioritize test cases according to some criterion, for example fastest rate of code coverage or most frequently used features first.Elbaum et al. report what they call an 'embedded multiple case study' to ascertain how the size, distribution and location of modifications made to software affect the effectiveness and efficiency of four test case selection and four test case prioritization techniques.Four non-trivial C programs, whose modification histories were available, were used as case studies.Since regression faults were not available, faults were seeded that were as realistic as possible and involved code deleted from, inserted into or modified between versions.By careful study of the trends in the results across all four programs, Elbaum et al. have gleaned five separate implications for regression testers and confirmed that change attributes play a significant role in the performance of regression testing.