Systematic evaluation of test failure results
Bertrand Meyer, Ilinca Ciupa, Manuel Oriol, Andreas Leitner, Raluca Borca-Mureşan · 2007
Systematic software testing provides an important source of software failure analysis. The field suffers, however, from insufficiently reproducible results, lack of standard credible data, and insufficiently explicit assumptions. The present article attempts to provide an objective basis for failure analysis through an automatic testing framework (AutoTest) for contractequipped software. We present five principles for scientific failure analysis, a set of reproducible test results, and a first analysis of their consequences for software development. 1 Testing strategies and their purpose The area of software testing provides one of the best possible illustrations of the lack of credible large-scale failure analysis highlighted by the call for proposals for this workshop. While research in software testing has made considerable advances in recent years, and succeeded in rehabilitating an approach that used to arise considerable suspicions caused by the proponents of formal software development, it still suffers from insufficient credibility of its results. In particular: • There are several criteria for assessing the effectiveness of testing strategies, such as number of faults uncovered, number of tests to first fault, time to first fault uncovered, code coverage, ratio of faultrevealing test cases out of all test cases run. None has achieved universal acceptance.