DPRFuzz: Enhancing Vulnerability Mining With Two-Stage Reinforcement Learning
Liqun Yang, Ruihao Li, Chaoren Wei, Jian Yang, Yuze Yang, Liang Sun, Dong Ya Zhao, Zhoujun Li · IEEE Transactions on Industrial Informatics · 2025
American fuzzy lop (AFL), as a representative tool for fuzzing, is capable of uncovering security vulnerabilities in industrial systems. It suffers from consuming a large amount of computational resources during the mutation. To improve the performance of AFL, researchers adopt algorithms, such as particle swarm optimization and long short-term memory, to optimize mutation operator selection. However, challenges persist in these approaches integrated with AFL, including optimization model complexity, insufficient accuracy, and poor generalization scalability. To address these issues, the article proposes a new fuzzer calledDPRFuzzto optimize AFL’s mutation phases. First, in the deterministic mutation strategy mutation phase, deep Q network and trust region policy optimization are leveraged to precisely generate effective mutated samples through perceiving mutation process in a relatively short time. Then, to boost the efficiency of the Havoc random mutation phase, we improve the Thompson sampling algorithm based on a multiagent strategy to generate an overall optimal mutation strategy chain. Finally, the approach is tested on eight programs, such asreadelf,tcpdump,andnm, and the advantages ofDPRFuzzare analyzed. Most importantly, the experiment reveals results thatDPRFuzzachieves better fuzzing performance compared to the traditional and other AFL-based fuzzers, such as AFL, AFL++, AFLSmart, etc. On average,DPRFuzzachieves an improvement in code coverage of over 10%, along with a significant increase in the number of crashes.