Empirical foundations for network defense
Paul Barford, Vinod Yegneswaran · 2006
The increasing sophistication and prevalence of attacks and intrusions on the Internet demands a better understanding of the nature of this activity and a fundamental re-thinking of network defense systems. This thesis describes results from a four year longitudinal investigation of Internet attack activity and our pursuit of insights into its patterns, diversity and evolution. The study builds the foundation for an empirical approach to protecting networks that aims to detect attacks in a scalable, accurate and timely manner. Our methodology follows two broad themes. We use measurements as a means to refine performance of existing systems and we incorporate automation and adaptivity in our system design. This enables defense systems to continuously learn from past history and dynamically respond to emerging threats. In the first part of the thesis, we describe a series of measurement studies using attack data collected from a diverse set of networks. They include analysis of a repository of daily firewall and NIDS logs obtained from over 1600 production networks and data collected at an unused but routable/8 network (a contiguous segment of 224 IP addresses or 1/256 of Internet's address space). Our measurement results highlight the importance of data-sharing across networks and its ability to improve perspective on attack activity. They also demonstrate the benefits of monitoring unused networks, and the utility of responding to traffic observed at unused networks as a means to classify (and infer intent of) anomalous traffic. Finally, they shed light on the inherent diversity of attack traffic, its temporal characteristics and variability across networks. These measurement results guide the development of our network defense systems. Our systems include capabilities for efficient monitoring, analysis and detection of attacks at both used and unused networks in the Internet. The monitoring system, which we call the Internet Sink (ISink), has the capability to simulate several popular network services and scalably monitor large telescopes. Key aspects of iSink's design include its use of stateless monitors and development of traffic filtering schemes to reduce data volume. The analysis system (NetSA) efficiently summarizes traffic collected at iSinks, has the capability to isolate both novel and large-scale events of interest, and implements adaptivity by using a database to continuously track past history. The detection system, which we call Nemean, implements learning algorithms that automatically build signatures from iSink data, which can be used to protect production networks. Nemean's design emphasizes a protocol semantics-aware approach for building signatures that are resilient to false-positives. Our results demonstrate that signatures generated by Nemean for NetBIOS and HTTP have detection rates comparable to popular open-source NIDS (over 99%) with a 0% false positive rate. The final component is a network address shuffling system called Kaleidoscope, that addresses the problem of protecting our sensors from attackers who attempt to actively build maps of such network monitors.