Detecting Malicious Use with Unlabelled Data Using Clustering and Outlier Analysis
G. S. Knight, L. Carosielli · 2003
Most commercial intrusion detection systems (IDSs) presently available are signature-based network IDSs. Organisations using these IDSs are still experiencing difficulties in detecting intrusive activity on their networks since novel new attacks are consistently being encountered, and analysts can miss legitimate alarms when reviewing large alarm logs that contain a high number of false positives. There has been research investigating the use of data mining techniques to effectively detect malicious activity in an enterprise network. The results of many of these projects have demonstrated that these techniques can be effective when trained/calibrated using labelled datasets. Labelled datasets identify and characterize normal and malicious traffic for use in training/calibrating the detection sensor. However, the creation of labelled datasets is resource intensive. It requires a significant effort by security analysts to create a data set that characterises the traffic in a specific enterprise network environment. This research simulates and analyses malicious activity on an enterprise network to explore the detection of malicious activity with data mining techniques using unlabelled datasets. Semi-discrete decomposition (SDD) is used as a clustering and outlier analysis technique to characterize network traffic as either normal or anomalous.