Unsupervised anomaly detection on log-based time series data
Oskar Granlund · KTH Publication Database DiVA (KTH Royal Institute of Technology) · 2019
Due to a constant increase in the number of connected devices and there is an increased demand for confidentiality, availability, and integrity on applications. This thesis was focused on unsupervised anomaly detection in data centers. It evaluates how suitable open source state-of-the-art solutions are at finding abnormal trends and patterns in log-based data streams. The methods used in this work are Principal component analysis (PCA), LogCluster, and Hierarchical temporal memory (HTM). They were evaluated using F-score on a real data set from an Apache access log. The data set was carefully chosen to represent a normal state in which close to no anomalous events occurred. Af- terward, 0.5% of the data points were transformed into anomalous data points, calculated based on the average frequency of log events matching a certain pattern. PCA showed the best performance with an F-score ranging from 0.4 - 0.56. The second best method was LogCluster but the HTM methods did not show adequate results. The result showed that PCA can find approximately 50% of the injected anomalies, this can be used to improve the confidentiality, integrity and availability of applications.