Better Language Models of Code through Self-Improvement
Hung Quoc To, Nghi Bui, Jin Guo, Tien N. Nguyen · 2023
Pre-trained language models for code (PLMCs) have gained attention in recent research.These models are pre-trained on large-scale datasets using multi-modal objectives.However, finetuning them requires extensive supervision and is limited by the size of the dataset provided.We aim to improve this issue by proposing a data augmentation framework using knowledge distillation.Our framework utilizes knowledge gained during the pre-training and finetuning stage to augment training data, which is then used for the next step.We incorporate this framework into the state-of-the-art language models, such as CodeT5, CodeBERT, and UnixCoder.The results show that our framework significantly improves PLMCs' performance in sequence-generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.* Equal contribution.Listing order is based on the alphabetical ordering of author surnames.