Applying Knowledge-Guided Deep Reinforcement Learning with Graph Neural Networks for Compiler Optimization
Zujie Li, Huabiao Qin, Yixiang Xie · 2025
Compilation flags optimization is a key technique in program performance enhancement, aiming to automatically optimize the execution order and number of passes, ultimately producing the optimal combination of optimization passes for best performance. Traditional research predominantly employs iterative compilation, heuristic algorithms and machine learning approaches. However, the insufficient representation of the program makes it difficult to discover the optimal combinations of optimization passes and the vast search space results in low search efficiency. To address these issues, this paper models the selection process of optimization passes as a Markov decision process and proposes an optimization passes selection method based on deep reinforcement learning (RL), automating the generation of optimization passes for various programs and improve execution efficiency. Firstly, a hybrid representation approach is designed by graph neural networks to enhance the program feature representation, which combines static program information and the structure features extracted from program control flow graphs. Secondly, we propose a novel optimization strategy which considers that the given extensive number of optimization passes may lead to prolonged RL training in large action spaces. We leverage the prior knowledge derived from intermediate representation of optimization structure to guide the decision-making of the RL model, thereby accelerating model convergence. Experimental result shows that, evaluated on benchmarks with execution time as the performance metric, the proposed method achieves up to a 2.82× performance improvement compared to heuristic algorithms and a 2.06× improvement over other RL methods.