Machine Learning-Based Intrusion Detection Systems: Capabilities, Methodologies, and Open Research Challenges
Chaoyu Zhang, Ning Wang, Y. Thomas Hou, Wenjing Lou · 2025
Intrusion Detection Systems (IDSs) are vital for protecting computer networks against unauthorized access and evolving cyber threats. Traditional signature-based IDSs, while effective for known attacks, struggle to identify novel and sophisticated threats. Machine Learning-based Intrusion Detection Systems (ML-IDSs) have emerged as a promising solution, offering enhanced capabilities for threat detection, adaptability, and response. This chapter explores the promising capabilities of ML-IDSs in cybersecurity, focusing on five key strengths. First, machine learning (ML) algorithms enable the detection of zero-day attacks by identifying anomalous patterns in network traffic without the need for predefined signatures. Second, explainable AI (XAI) integrated with ML models enhances understanding and trust by elucidating the decision-making process behind alerts. Third, ML methods facilitate the detection of intrusions in encrypted traffic, maintaining data privacy while ensuring security. Fourth, the use of Graph Neural Networks (GNNs) introduces context-aware and provenancebased detection, providing structured analysis of relationships within network data. Finally, Large Language Models (LLMs) enhance IDSs by understanding network traffic as human language, detecting complex intrusions, and processing sequential network data for intrusion detection. This paper highlights these capabilities and presents an organized analysis of the stateof-the-art contributions in each area, followed by a detailed discussion of open research challenges and future directions in this important domain. This comprehensive review underscores the transformative role of ML in IDSs in addressing emerging and increasingly powerful cyber threats.