Enhanced Intrusion Detection in Big Data Systems: A Machine Learning Approach
Balachandar Paulraj · 2024
Robust intrusion detection systems (IDS) are critical in big data environments due to the growing volume and complexity of network traffic, as they protect against cyberattacks. Conventional signature-based intrusion detection systems have difficulty keeping up with new attack techniques. In large data network security, machine learning and deep learning methods are viable substitutes for anomaly detection. To attain better performance on big data network traffic datasets, this study suggests a unique deep learning-based intrusion detection system (IDS) that makes use of capsule networks, bi-directional LSTMs (BiLSTMs), and embedding layers. We create a deep learning model for feature extraction and pattern detection from network traffic data by combining an embedding layer, a capsule layer, and a BiLSTM layer. To test the model’s efficacy in intrusion detection, two benchmark datasets UNSW-NB15 and an IoT dataset are used. With a test accuracy of 99.99% on the UNSWNB15 dataset, the suggested model performs remarkably well on both datasets. This shows the model’s greater capacity to detect anomalies and possible intrusions within big data network traffic, surpassing the accuracy reported in earlier studies. The results demonstrate how well the suggested deep learning architecture for big data intrusion detection works. Due to its excellent accuracy and generalizability over a wide range of datasets, the model has practical applications in protecting big data networks from cyberattacks.