Development of a network intrusion detection system using Apache Hadoop and Spark
Keisuke Kato, Vitaly V. Klyuev · 2017
Cyber attacks on network communication are executed against companies, governments, and even individuals. A number of these attacks is drastically increased over the last decade. Nowadays, protecting private data, latest research data, etc. is a crucial problem. Therefore, developing an intelligent system to detect the attacks is required. In this paper, we propose an anomaly based network intrusion detection system. The system is capable to analyze huge datasets in a short period of time. We utilized 90.9 GB of a real network packet dataset provided by the Information Security Centre of Excellence at the University of New Brunswick. The system analyzes the packet capture files of this dataset in the environment by using Apache Hadoop and Spark. An approach to implement the system is based on Hive SQL and unsupervised learning algorithms. The accuracy of the proposed detection system is 86.2% with 13% of the false positive rate. These results are promising to detect attacks in real-time.