A Robust Residual Autoencoder Framework for Anomaly-Based Network Intrusion Detection
Nouman Ijaz, Sana Ullah Jan, Insoo Koo · 2026
The growing complexity of network environments and the rapid emergence of zero-day attacks have undermined the effectiveness of traditional signature-based intrusion detection systems (IDS). To address these limitations, this paper proposes an autoencoder framework, termed a Residual Autoencoder (RAE), for unsupervised anomaly-based intrusion detection. The proposed model is trained solely on benign traffic and detects abnormal behavior by computing reconstruction errors on unseen data. This is achieved by adding a residual skip connection to maintain information of low-level features and stabilize gradient flow during training. Furthermore, a hybrid loss, which is a combination of Smooth L1 loss and cosine similarity, simultaneously applies numerical precision and structural consistency between the input and the reconstructed features. The experiments using the UNSW-NB15 dataset show that the proposed RAE is an effective framework to differentiate between normal and attack traffic, with an ROC-AUC score of 0.903, surpassing baseline methods, including the plain autoencoder (0.884), isolation forest (0.860), and one-class SVM networks (0.820). The results indicate that the model can effectively handle new and zero-day attacks. Additionally, it offers a robust, data-efficient, and scalable solution for next-generation network intrusion detection systems.