Incremental outlier detection in data streams using local correlation integral
Xinjie Lu, Tian Yang, Zaifei Liao, Manzoor Elahi, Wei Liu, Hongan Wang · 2009
In this paper, an incremental outlier detection technique capable of dealing with a large amount of data is presented and evaluated in the context of intrusion detection. The proposed method is based on the LOcal Correlation Integral (LOCI for short). The detection technique consists of two parts. The first part named insertion receives the sequence of input point and updates Multi-granularity DEviation Factor (MDEF) of the point at intervals. The second part named deletion deletes one or a batch of points. This technique is able to process streaming data in a single scan. Moreover, the number of updates in the incremental LOCI algorithm per insertion/deletion of a single data record does not depend on the total number of data records. Experimental results with real life data sets show that the technique is capable of dealing with data streams, successfully detecting outlier.