Functional Overlap Reranking for Neural Code Generation
Hung Quoc To, Le-Minh Nguyen, Nghi Bui · 2024
Code Large Language Models (CodeLLMs) have ushered in a new era in code generation advancements.However, selecting the best code solutions from all possible CodeLLM outputs remains a challenge.Previous methods often overlooked the intricate functional similarities and interactions between solution clusters.We introduce SRank, a novel reranking strategy for selecting the best solutions from code generation, focusing on modeling the relationships between clusters of solutions.By quantifying the functional overlap between solution clusters, our approach provides a better ranking strategy for code solutions.Empirical results show that our method achieves remarkable results on the pass@1 score.For instance, on the Human-Eval benchmark, we achieve 69.66% in pass@1 with Codex002, 75.31% with Wiz-ardCoder, 53.99% with StarCoder, and 60.55% with CodeGen, surpassing state-of-the-art code generation reranking methods such as CodeT and Coder-Reviewer on the same CodeLLM by a significant margin (≈ 6.1% improvement on average).Even in scenarios with a limited number of sampled solutions and test cases, our approach demonstrates robustness and superiority, marking a new benchmark in code generation reranking.Our implementation can be found at https://github.com/ FSoft-AI4Code/SRank-CodeRanker.tilingual training for software engineering.In Proceedings of the 44th