RLHavoc:Enhanced Historical Data Orientation Enhances Code Coverage
Shihe Zhang, Chaohao Liao, Renhua Liu · 2023
Fuzzing is a common method in software test engineering, which tries to trigger potential vulnerabilities in the target program by continuously providing modified input to the test target. The traditional fuzzy test tools are white box test, gray box test, black box test. The mutation strategies used in these traditional testing tools have high randomness. As a result, the traditional test methods have high randomness in the selection of strategic actions, so effective strategic actions maybe be ignored. Aiming at improving the coverage rate of fuzzy test code based on variation, we implement RLHavoc to improve the efficiency of American fuzzy lop(AFL) based on Deep Q Network(DQN) algorithm by using DQN to prioritize the generation of test cases in AFL. RLHavoc combines AFL’s Havoc phase mutation strategy with DQN algorithm. DQN algorithm takes advantage of AFL’s Havoc strategy to conduct grey box testing for different binaries and use comprehensive historical data to guide the action selection of variation strategies in the fuzzy test vulnerability detection process, taking seeds as state and new the number of unique execution paths as reward, which enhances the action space orientation of the secondary fuzz target binaries, thus improving the code coverage. Compared to the traditional AFL fuzz effect, our experimental results show that the code coverage rate is improved by about 3%.