Are Deletion Mutants Easier to Identify Manually?
Vinícius H. S. Durelli, Nilton Mendes de Souza, Márcio Eduardo Delamaro · 2017
Mutation testing changes a program by applying mutation operators, which generate many slightly different versions of the original program (i.e., mutants). Identifying which mutants are equivalent to the original program is one of the major barriers to a wider adoption of mutation testing. Recent studies suggest that an effective approach to mitigating this problem is to use only a subset of all mutation operators: some operators result in few mutants, but these mutants yield tests that are practically as effective as using all mutants. One research direction has been to investigate the use of only one type of operator: deletion mutation operators. Deletion operators are deemed to be effective and, based on an intuitive notion, we conjectured that these operators lead to mutants that are easier to manually assess for equivalence. Thus, we set out to compare the (human) cost of detecting the equivalence of mutants generated by deletion operators with the cost of analyzing mutants created by traditional operators. To this end, we designed an experiment and conducted it with 12 graduate students. This study investigated two proxies for the cost of identifying equivalent mutants by hand: (i) number of mistakes made by the subjects and (ii) the length of time spent analyzing mutants. According to the results of our experiment, there is no significant difference between the costs of manually assessing traditional and deletion operators for equivalence. We also probed into the cost of each operator and found that CCDL is the most costly deletion operator. Another core finding is that ORRN, ORLN, and ORSN account for the lion's share of the cost involved in manually assessing traditional mutants for equivalence.