Retrieval-Augmented Fine-Tuning for Improving Retrieve-and-Edit Based Assertion Generation
Hongyan Li, Weifeng Sun, Meng Ling Yan, Ling Xu, Qiang Li, Xiaohong Zhang, Hongyu Zhang · IEEE Transactions on Software Engineering · 2025
Unit Testing is crucial in software development and maintenance, aiming to verify that the implemented functionality is consistent with the expected functionality. A unit test is composed of two parts: a test prefix, which drives the unit under test to a particular state, and a test assertion, which determines what the expected behavior is under that state. To reduce the effort of conducting unit tests manually, Yu et al. proposed an integrated approach (integrationfor short), combining information retrieval with a deep learning-based approach to generate assertions for test prefixes, and obtained promising results. In our previous work, we found that the overall performance ofintegrationis mainly due to its success in retrieving assertions. Moreover,integrationis limited to specific types of edit operations and struggles to understand the semantic differences between the retrieved focal-test (focal-testincludes a test prefix and a unit under test) and the input focal-test. Based on these insights, we then proposed a retrieve-and-edit approach namedEditAS to learn the assertion edit patterns to improve the effectiveness of assertion generation in our prior study. Despite being promising, we find that the effectiveness ofEditAS can be further improved. Our analysis shows that: ① The editing ability ofEditAS still has ample room for improvement. Its performance degrades as the edit distance between the retrieval assertion and ground truth increases. Specifically, the average accuracy ofEditAS is 12.38% when the edit distance is greater than 5. ②EditAS lacks a fine-grained semantic understanding of both the retrieved focal-test and the input focal-test themselves, which leads to many inaccurate token modifications. In particular, an average of 25.57% of the incorrectly generated assertions that need to be modified are not modified, and an average of 6.45% of the assertions that match the ground truth are still modified. Thanks to pre-trained models employing pre-training paradigms on large-scale data, they tend to have good semantic comprehension and code generation abilities. In light of this, we proposeEditAS2, which improves retrieval-and-edit based assertion generation through retrieval-augmented fine-tuning. Specifically,EditAS2first retrieves a similar focal-test from a predefined corpus and treats its assertion as a prototype. Then,EditAS2uses a pre-trained model, CodeT5, to learn the semantics of the input and similar focal-tests as well as assertion editing patterns to automatically edit the prototype. We first evaluate theEditAS2for its inference performance on two large-scale datasets, and the experimental results show thatEditAS2outperforms state-of-the-art assertion generation methods and pre-trained models, with average performance improvements of 15.93%-129.19% and 11.01%-68.88% in accuracy and CodeBLEU, respectively. We also evaluate the performance ofEditAS2in detecting real-world bugs from Defects4J. The experimental results indicate thatEditAS2achieves the best bug detection performance among all the methods.