Automated Scalable Platform for Packet Traffic Analysis
Miguel José Cavadas Santos · Open Repository of the University of Porto (University of Porto) · 2016
In these days a growing rate of cybercrime exists, in both number and complexity of the attacks that are being made. This crime is becoming more powerful, persistent, costly and focused on targets by the day. Preventing and detecting this dangerous situation is key to protect any Internet based company, and, because of this, security best practices are followed, firewalls are implemented, and very powerful Intrusion Detection Systems and Intrusion Prevention Systems are developed. All types of IDS have their particular strengths but not a single one is perfect: they can fail to detect some specially crafted attacks, not have enough processing speed to deal with the threat or even assume a certain connection to be an attack when it is not. There are two major types of IDS signature and anomaly based and much work is constantly being done to enhance them. In this thesis we studied Snort, a very popular free signature-based IDS. Most of the state of the art solutions found for Snort address issues like rule inefficiency, improving detection capability, using big data analytics on the alerts, and studying the false positive ratio. No work was found that tackles the performance issue when too many rules are loaded into Snort, or as it is also called, the Security over Connectivity problem. There is also a lack of studies about the integration of Snort processing with MapReduce jobs. With that in mind, the goal of this thesis is to study the impact of distributing the processing of Snort through a computer cluster and compare the results obtained, such as times measured and alerts produced, with the state of the art solutions. To accomplish this we developed The Hunter, a completely automated and scalable platform that distributes packet captures for Snort to process in an already deployed computer cluster, while also allowing integration with Hadoop and MapReduce. The results obtained with the developed tool confirmed that a distributed approach to Snort can indeed coexist with Hadoop in its core, while maintaining a fast processing time and good detection rate.