BePilot: An AI Programming Assistant for Compiler Backend Development
Ming Zhong, Xin Sun, Fang Lv, Lulin Wang, Hongna Geng, Lei Qiu, Huimin Cui, Xiaobing Feng · ACM Transactions on Software Engineering and Methodology · 2025
Compiler backends are tasked with generating executable machine code for various processors. As the diversity of processors continues to grow, it is imperative for programmers to tailor specific compiler backends to accommodate each one. However, compiler backend development remains a labor-intensive and time-consuming process, with limited automation tools available. Although large language models (LLMs) have demonstrated strong abilities in code completion and code generation tasks, the lack of appropriate datasets for compiler backend development limits the application of LLMs in this field. In this article, we introduce ComBack++, a multilingual dataset covering C/C++, machine description, and TableGen, with 184 backends from GCC and LLVM, four backend-specific tasks. Based on ComBack++, we present BePilot, a compiler backend-specific LLM available in two sizes: BePilot-1.5B and BePilot-7B. We also introduce CB-Retriever , a retriever that constructs few-shot prompts via in-context learning to improve vanilla LLM performance in resource-constrained settings. Experimental results show that BePilot-1.5B and BePilot-7B achieve significantly higher accuracy across four tasks in ComBack++ compared to 12 baseline LLMs (125M–34B parameters). In addition, CB-Retriever consistently boosts the accuracy of six mainstream LLMs. Both BePilot-1.5B and BePilot-7B, as well as vanilla LLMs augmented with CB-Retriever , outperform the traditional manual compiler backend development approach (Fork-Flow) in efficiency across all four tasks in ComBack++. Furthermore, human evaluation by four experienced compiler backend developers confirms that BePilot not only improves development efficiency over Fork-Flow but also surpasses commercial AI programming assistants such as GPT-4o-mini and Gemini2-Flash in terms of code quality. These findings confirm that BePilot and CB-Retriever can substantially enhance compiler backend development efficiency.