On the Coupling between Vulnerabilities and LLM-Generated Mutants: A Study on Vul4J Dataset
Aayush Garg, Renzo Degiovanni, Mike Papadakis, Yves Le Traon · 2024
With the release of powerful language models trained on large code corpus (e.g., CodeBERT, trained on 6.4 million programs), a new family of mutation testing tools has arisen that promises to generate more “natural” mutants, where the mutated code aims at following the implicit rules and coding conventions produced by programmers. In this paper, we empirically study the observable behavior of CodeBERT-generated mutants and to what extent are these coupled with software vulnerabilities. To do so, we carefully analyze 45 reproducible vulnerabilities from the Vul4J dataset to determine whether the mutants and vulnerabilities fail the same tests and whether the failures are for the same reasons or not. Hence, we define different degrees of vulnerability-coupling classes. Strongly coupled mutants fail the same tests for the same reasons as the vulnerabilities, while test coupled mutants fail the same tests but for some different reason as the vulnerabilities. Partial coupling classes are also considered. Overall, CodeBERT-generated mutants strongly coupled with 32 out of these 45 vulnerabilities (i.e. The mutants fail on the same tests for the same reasons), while another 7 vulnerabilities are test-coupled by CodeBERT mutants (i.e. The mutants fail on the same tests but not for the same reasons). Interestingly, CodeBERT mutants are diverse enough to couple vulnerabilities from 14 out of the 15 types of vulnerabilities explored, i.e., CWEs (Common Weakness Enumeration). Finally, we observe that strongly coupled mutants are scarce (1.17 % of the killable mutants), test coupled mutants represent 7.2 %, and 64.9 % of the killable mutants are not coupled with the vulnerabilities.