Detecting anomalies in system log files using machine learning techniques
Tim Zwietasch · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · 2014
Log files, which are produced in almost all larger computer systems today, contain highly valuable information about the health and behavior of the system and thus they are consulted very often in order to analyze behavioral aspects of the system. Because of the very high number of log entries produced in some systems, it is however extremely difficult to find relevant information in these files. Computer-based log analysis techniques are therefore indispensable for the process of finding relevant data in log files. However, a big problem in finding important events in log files is, that one single event without any context does not always provide enough information to detect the cause of the error, nor enough information to be detected by simple algorithms like the search with regular expressions. In this work, three different data representations for textual information are developed and evaluated, which focus on the contextual relationship between the data in the input. A new position-based anomaly detection algorithm is implemented and compared to various existing algorithms based on the three new representations. The algorithms are executed on a semantically filtered set of a labeled BlueGene/L log file and evaluated by analyzing the correlation between the labels contained in the log file and the anomalous events created by the algorithms. The results show, that the developed anomaly detection algorithm generates the most correlating set of anomalies by using one of the three representations.