Big Data Cybersecurity Monitoring System using Machine Learning
Rahul Pramod Krupani, M. Srinivasa Aditya, C S Prithvi Raghavan, H S Gururaja · 2021
The rapid growth of the Internet, and other technological advances in recent times have led to generation of data at an alarming rate. Big Data Analytics (BDA) is the process of analyzing the large amounts of data generated to uncover information to make business decisions. In addition, the wide variety of applications of this data has made it a very high value target, making cybersecurity increasingly important, while traditional methods fail to keep up with the large scale of data. Our aim is to build an architecture dedicated to medium scale enterprise networks which can be potentially resized to high scale networks and their security monitoring. This Big data monitoring system will use DNS data, NetFlow records, and HTTP traffic to train ML models to analyze and correlate the data, allowing us to detect any type of threat using well-known Big Data frameworks like Spark.