Effectiveness of Inadequate Test Suites : A Case Study of Mutation Analysis
Hikari Watanabe · KTH Publication Database DiVA (KTH Royal Institute of Technology) · 2017
How can you tell whether your test suites are reliable? This is often done through the use of coverage criterion that would define a set of requirements that the test suites need to fulfill in order to be considered reliable. The most widely used criterion is those referred to as code coverage, where the degree to which the code base is covered is used as an arbitrary measure of how good the test suites are. Achieving high coverage would indicate an adequate test suite i.e. reliable according to the standards of code coverage. However, covering a line of code does not necessarily mean that it has been tested. Thus, code coverage can only tell you what parts of the code base have not been tested, opposed to what have been tested. Mutation testing on the other hand is an approach to evaluate the adequacy of test suites through their fault detection ability, rather than how much of the code base they cover. This thesis performs mutation analysis on a project with inadequate code coverage. The present testing effort on unit level is evaluated and the cost and benefits of adopting mutation testing as a testing method is evaluated.