Memory-efficient anomaly detection for online data streams
Shiming He, Chenxi Guo · 2024
Network Intrusion Detection refers to the use of anomaly detection to protect network security, that is, detecting and identifying anomalies and malicious behavior in the network by monitoring network traffic and other metrics. Identifying anomalies and intrusions plays a crucial role in network security, as it helps organizations to timely detect and respond to various network attacks and threats. Unlike static data, intrusion detection data is usually in the form of data streams. However, existing methods have not fully considered the inherent characteristics of data streams, such as infinity, real-time nature, and concept drift, which leads to lower detection accuracy and significant memory waste. Considering these issues, we propose an online anomaly detection method called MEO-AD, based on Locality Sensitive Hashing (LSH), Isolation Forest, and an adaptive updating. It can handle the concept drift problem of data streams. We evaluate MEO-AD on two public network intrusion detection datasets. Experimental results demonstrate the accuracy of the proposed method and its lower memory consumption. We consider three types of LSH. The LSH with Euclidean distance presents the best performance.