AI-Based Algorithm for Zero-Day Attack Detection Using Reinforcement Learning
Ola Nasir, Waleed Amer, Ala Hamarsheh, Ali Mahmoud Ali, Amani Abu Zaid · 2025
Zero-Day attacks represent one of the most challenging threats in modern cybersecurity, as they exploit unknown vulnerabilities and bypass traditional detection systems. In this paper, we propose a hybrid threat detection system that integrates three key components: a Random Forest-based supervised classification model for detecting known attack patterns, an autoencoder-based anomaly detection module to uncover novel threats, and a Reinforcement Learning (RL) agent to dynamically optimize response strategies in real-time. The proposed system achieves a detection Accuracy of 97.8% for known attacks. The autoencoder, trained exclusively on normal traffic, employs a dynamically determined threshold (set at 0.8409) to flag anomalous behavior indicative of Zero-Day attacks. Furthermore, the RL agent adapts quickly to evolving threats, effectively filtering out 53.2% of benign traffic while blocking 39.7% of attack traffic and quarantining 7.1% of ambiguous cases for further inspection. Experimental results demonstrate that this integrated approach not only enhances top-level threat prevention but also significantly reduces false alerts.