Log Anomaly Detection with a Frequency-based Encoding Approach
Lu Nong, Tao Li, Aiqun Hu · 2024
Software systems often use logs to record runtime information, facilitating the identification and counter of runtime faults or intrusions. Numerous automatic log analysis tools have emerged, which can well extract log templates that represent the common structure of logs from log set, thus promoting research studies of log anomaly detection. However, existing studies on log template-based methods fail to consider: a) unstable semantic analysis due to words in log templates exceeding the vocabulary, and b) the occurrence frequency of log templates. In this work, we propose a log encoding method based on letter frequency, preserving semantic information while addressing vocabulary limitation. Additionally, we consider template occurrence frequency to enhance information carried by log template vectors. We evaluate the effectiveness of our approach through experiments on standard dataset. The result demonstrates that the model performs well in both recall and precision.