FSMFLog: Discovering Anomalous Logs Combining Full Semantic Information and Multifeature Fusion
Weina Niu, Zimu Li, Zhaoxu He, Aduo Wang, Beibei Li, Xiaosong Zhang · IEEE Internet of Things Journal · 2023
Industrial Internet of Things devices usually use log information to record their runtime status, so log-based anomaly detection can contribute to discovering device failures in time. The first step of log-based anomaly detection is log parsing. However, existing methods mainly extract log templates for analysis, which ignore some words that represent key semantics. Such omissions may cause semantic misunderstandings and further affect the performance of anomaly detection. On the other hand, existing deep learning-based log anomaly detection approaches only consider the sequential relations among log messages, ignoring the log time and type information. In this article, we propose an anomaly detection method called FSMFLog based on full semantic information and multifeature fusion. FSMFLog uses log word lists instead of log templates to represent semantic information. Specifically, the variable part is first removed through preprocessing, and then the log sentences are initially clustered using two heuristic strategies, after which the words in the log content are clustered through the prefix tree structure. By integrating semantic features, time features, and type features, FSMFLog also trains a bidirectional GRU model based on an attention mechanism. Evaluation on 16 real-word log data sets from LogHub shows that FSMFLog achieves a higher log parsing accuracy, outperforming other five state-of-the-art log parsing methods. We also evaluated FSMFLog on two most widely used public data sets (HDFS and BGL), and the results demonstrate the effectiveness of FSMFLog, outperforming the compared approaches using deep learning with an average increase of more than 10% in$F1$-score.