Multi-Agent Deep Q-Network Based Ant Colony Optimization for Advanced Network Intrusion Detection
Amani Bacha, Farah Barika Ktata, Olfa Belkahla Driss · 2025
The growing complexity and interconnectivity of modern systems have made network security and anomaly detection a critical challenge in sectors such as defense, academia, finance, and IoT-enabled infrastructures. Traditional rule-based methods often fail to adapt to the dynamic and evolving nature of cyber threats, especially in IoT environments. These systems are inherently complex due to the large volume and variety of data generated by interconnected devices. To address these challenges, we propose a novel hybrid framework combining Multi-Agent Reinforcement Learning (MARL) with Ant Colony Optimization (ACO) to enhance anomaly detection in dynamic and large-scale networks. ACO, widely recognized for its feature extraction capabilities, guides Deep Q-network-based agents within a distributed MARL system. Each agent operates independently, integrating Q-values and pheromone signals that encode optimal actions based on historical rewards. This mechanism fosters implicit coordination among agents, addressing challenges such as non-stationarity in state-action spaces. The framework also ensures scalability and data privacy, making it particularly suitable for IoT systems where real-time responsiveness and confidentiality are essential. Our framework is evaluated on the NSL-KDD dataset, a widely used benchmark for intrusion detection. In binary classification, it reaches nearly 98 percent accuracy, with precision, recall and F1-score also close to 99 percent. In the multi-class setting, it achieves over 80 percent accuracy, with high recall and an F1-score close to 89 percent surpassing comparable state-of-the-art approaches and demonstrating its practical advantages. These results confirm not only the system's robustness and adaptability in managing real-world cyber threats but also its ability to support decision traceability.