An Analysis and Comparison of Mutation Testing Tools for Python
Kadiatou Diallo, Zizhao Chen, W. Eric Wong, Shou-Yu Lee · 2024
Software testing is a crucial phase in the software development lifecycle, yet it often becomes a challenging task for engineers who must ensure comprehensive test coverage. While python unit testing frameworks like pytest and unit test are widely used to validate code functionality, passing these tests does not necessarily imply meaningful or effective testing. Mutation Testing addresses this gap by introducing deliberate faults, known as mutants, into the program under test to assess the quality of its test suite. This study evaluates five Python-based mutation testing tools: MutPy, Mutmut, Mutatest, Poodle, and Cosmic Ray, by applying them to two different open-source programs. The tools’ performances were compared using mutation-specific and tool-specific criteria to determine their strengths, limitations, and analyze which tool created the most competent mutants. In the end, Poodle created the most competent mutants with a 50.9% competency score, Cosmic Ray came next with 25.7% competency score, and Mutmut was inconclusive due to incomplete mutation results.