Introduction to the special issue on Mutation Testing
Yue Jia, Mercedes G. Merayo, Mark Harman · Software Testing Verification and Reliability · 2015
It is our pleasure to introduce this special issue on Mutation Testing. The special issue contains nine papers, including four extended versions of papers presented at the 7th International Workshop on Mutation Analysis and five new submissions. We have divided the special issue into three broad areas based on the topics covered. The first area focuses on the techniques for making mutation testing more efficient and practical; the second area revisits some fundamental questions about mutants, whilst the third area presents some advanced applications of mutation testing for model-based testing. Mutation Testing has been proven to be an effective way to measure the quality of a test suite in terms of its ability to detect faults 1. The history of mutation testing can be traced back to 1971 in a publication by Richard Lipton 2 as well as in publications from the late 1970s by DeMillo et al. 3 and Hamlet 4. In Mutation Testing, faults are deliberately seeded into the original program (by simple syntactic changes) to create a set of faulty programs called mutants, each containing a different syntactic change. The general principle underpinning Mutation Testing is that artificial faults can be used to represent common programming mistakes. By carefully choosing the location within the program and the types of faults, it is possible to simulate any test adequacy criteria whilst providing improved fault detection. A recent survey on mutation testing provides evidence to suggest that the approach is increasing in maturity and practical application 5. One reason why mutation testing has become a popular testing approach is that it is a straightforward process to apply. To assess the quality of a given test set, the generated mutants are executed against the input test set. If the result of running a mutant is different from the result of running the original program for any test cases in the input test set, the seeded fault denoted by the mutant is detected. One outcome of the Mutation Testing process is the mutation score, which indicates the quality of the input test set. The mutation score is the ratio of the number of detected faults over the total number of seeded faults. Mutation Testing has been widely adopted in the academic community as a means to evaluate software testing techniques 6, as well as to generate tests and test oracles 7, 8. However, it still suffers from a number of problems that prevent the wider industrial uptake of this effective testing approach. One problem that prevents Mutation Testing from becoming a practical testing technique is the high computational cost of executing a large number of mutants against a test set. Other problems are related to the amount of effort involved in identifying equivalent mutants. Each submission received three reviews from a board of 36 mutation testing experts. For all submissions extended from the mutation workshop, we have recruited at least one new reviewer to ensure wider accessibility to a non-mutation expert testing audience. The first area covers the topic of making mutation testing more efficient and practical. The three papers in this area introduce novel techniques to optimize mutant execution, to reduce redundant mutants and to detect equivalent mutants. In the first paper 'Reducing Mutation Costs Through Uncovered Mutants', Pedro Reales Mateo and Macario Polo Usaola propose an improved mutant schema, namely, 'MUSIC' to reduce the execution cost for mutation testing. The MUSIC approach records runtime information about structural and mutation coverage; it reduces the execution cost by removing mutant execution tasks, which are not covered by the test cases. The second paper 'Higher Accuracy and Lower Run Time: Efficient Mutation Analysis using Non-redundant Mutation Operators' by René Just and Franz Schweiggert attempts to reduce the number of mutants by applying only non-redundant mutation operators. The authors identified a set of operators that tend to not generate any redundant mutants, and their results show that 20% of the runtime cost could be saved using the selected operators. The third paper 'Employing Second-order Mutation for Isolating First-order Equivalent Mutants' by Marinos Kintis, Mike Papadakis and Nicos Malevris seeks to automatically identify equivalent mutants through higher order mutation. Their approach combines impact analysis for both first-order and second-order mutants, and it achieved an equivalent mutant classification precision of 73% and a classification recall of 65%. The three papers in the second area revisit some fundamental questions about mutants and explore a new application of mutation testing. The first paper 'Quality Metrics for Mutation Testing with Applications to WS-BPEL Compositions' by Antonia Estero-Botaro, Francisco Palomo-Lozano, Inmaculada Medina-Bulo, Juan José Domínguez-Jiménez and Antonio García-Domínguez attempts to discover what it means for mutants to be effective. They formally define a set of metrics to measure the quality of mutation operators and evaluate them using WS-BPEL applications. The second paper 'MuRanker: a Mutant Ranking Tool' by Akbar Siami Namin, Xiaozhen Xue, Omar Rosas and Pankaj Sharma proposes metrics to measure the mutant complexity based on how easy or hard they are to kill. They implemented a prototype tool, MuRanker, which can help testers to prioritize the analysis of mutants based on their killing ability. The third paper 'Metallaxis-FL: Mutation-based Fault Localisation' by Mike Papadakis and Yves Le Traon explores the application of mutation testing for fault localization. This approach combines code coverage and mutation information to rank suspicious statements. The results show that it outperforms other traditional coverage-based fault detection approaches. The third area covers some advanced applications of mutation analysis for model-based testing techniques. This is an under-studied area compared with traditional program mutation. The first paper 'Using Mutation to Assess Fault Detection Capability of Model Review' by Paolo Arcaini, Angelo Gargantini and Elvinia Riccobene introduces a set of mutation operators for NuSMV Models. The mutant models simulate common behavioural faults and can be used to evaluate the fault detection ability of automated model review techniques. The second paper 'Towards an Automation of the Mutation Analysis Dedicated to Model Transformation' by Vincent Aranega, Jean-Marie Mottu, Anne Etien, Thomas Degueule, Benoit Baudry and Jean-Luc Dekeyser proposes to use mutation testing to test model transformations. They designed a set of mutation operators targeting three actions in model transformation: navigation, filtering and creation/modification. These are evaluated on the class2rdbms technique, which generates relational database management systems model from class diagrams. The third paper 'Model-based Mutation Testing from Security Protocols in HLPSL' by Frédéric Dadeau, Pierre-Cyrille Héam, Rafik Kheddam, Ghazi Maatoug and Michael Rusinowitch proposes a set of mutation operators to generate mutants for HLPSL security protocols. It also demonstrates that concretization test data generation techniques can be used to construct test scripts to kill the mutants. We wish to thank the authors and reviewers for their contributions to this special issue and Rob Hierons and Jeff Offutt for helping to manage the review process.