Evaluation of Machine Learning Intrusion Detection Systems Resiliency to Network Traffic Evolution
Florent Durécu, Maxime Puys, Gérard Chalhoub, Paul-Marie Grollemund · 2025
Research in the Network-based Intrusion Detection Systems (NIDS) field is a major focus for academics and the emergence of reliable and applicable detection models quickly became imperative for the integrity of critical infrastructures. Thus, Machine Learning (ML) models-specifically Deep Learning (DL) models- have been developed to detect network attacks and multiple training datasets are available, which allows to reproduce experiments. Nevertheless, the problem of intrusion detection is challenging since generating a representative set of training data is difficult due to the large variety of applications, and the fact that network infrastructures evolve over time. Also, a key concern often ignored is the different performances of models between experiments on known and often curated datasets, and in reality when deployed. In this paper, we propose a methodology to highlight the shortcomings of Anomaly-based NIDS, showcasing significant performance drops using realistic scenarios of evolution in the datasets. We implement a Denoising AutoEncoder (DAE) as a NIDS model. Then, we present a method to test and interpret the NIDS results based on analyzing the DAE's Mean Squared Error (MSE) to provide insights into the model's flow classifications. We advocate for considering the natural evolution of infrastructures when designing NIDS and highlight further research to address this challenge.