Knowledge discovery in computer network data: a security perspective
Carey E. Priebe, Kendall E. Giles · 2007
From a security perspective, computer network data is analyzed largely for two purposes: to detect known structures, and to identify previously unknown structures. As an example of the former, it is considered standard procedure to filter network traffic for previously identified viruses in order to prevent infection and to reduce virus spread. As an example of the latter, security researchers may want to search datasets in order to identify and discover previously unknown relationships or structures in the data, such as intrusions into a network by an external hacker. However, among other limitations, traditional methods of network data analysis are insufficient when processing large volumes of network traffic, do not allow for the discovery of local structures, do not visualize high-dimensional data in meaningful ways, and do not allow user input during search iterations. We present the development, analysis, and testing of a new framework for the analysis of network traffic data. In particular, and among others, the framework addresses the following questions: How is network traffic represented in high-dimensional space? (normalized graph Laplacians); How can we extract features from the data? (flow-based feature extraction); How to embed high-dimensional representations in low dimensions? (Laplacian Eigenmaps); How to intuitively visualize low-dimensional structures? (Fiedler Space projections); How to address scaleability concerns? (proximity computation, partitioning, and eigenpair computation approximations); How can the user be involved in the search? (iterative denoising applied to network data); How might this framework be used on empirical network data? (application to computer network intrusion data, backscatter data, and computer network application data). As such, this work presents theoretical as well as practical contributions, and these results are discussed within the context of traditional methods and techniques. Thus, due to its theoretical and applied benefits of visualizing and classifying a variety of unsupervised, heterogeneous, high-dimensional computer network traffic datasets, we feel that Iterative Denoising can be a unifying technology for protection, detection, and response groups coordinating around a network security monitoring system.