A Deep Reinforcement Learning Based Fuzzing Approach for Network Intrusion Detection Systems
Han Liu, Yifan Song, Qidi Jiao, Shuai Li, Fangfang Dang, Lijing Yan, Xiaorui Cui, Yu Li · 2025
As a critical infrastructure in modern cybersecurity architectures, Network Intrusion Detection Systems (NIDS) exhibit performance that is fundamentally constrained by the precision and comprehensiveness of their security rule sets. Current evaluation methodologies predominantly depend on static datasets or manually synthesized attack vectors, which fail to assess NIDS resilience against evolving threat landscapes. To enhance the intelligence and systematicness of strategy verification, this paper proposes a NIDS fuzz testing method based on deep reinforcement learning. This method uses the Deep Q-network (DQN) to model the fuzz testing process, realizing the dynamic selection of mutation strategies and optimizing the test case generation process, thereby achieving automated detection and improving detection efficiency. In this paper, a fuzz testing platform with Snort3 as the detection engine is constructed. A complete state space, action set and reward mechanism are designed. The testing process is abstracted as the Markov Decision Process (MDP), and experience replay and dual-network architecture are introduced to improve the stability and convergence speed of training. The experimental results show that the anomaly generation rate (AGR) of the proposed method reached 58%, which is 9.5 times higher than that of the traditional method (6.12%). In addition, the method exhibits fast convergence and sustained testing efficiency.