Research on Log Anomaly Detection Method Based on Topic Information
Shuai Li, Haiyang Liu, Dingding Li, Fangfang Dang, Ruimin Wang · 2023
During network communication, a large number of network devices generate massive data, recording the real-time operating status of the business. By collecting and analyzing logs, network failures have already occurred or potential failures can be discovered or predicted. However, log data usually has high dimensionality, irregularity, and massive volume, traditional rule-based or feature engineering-based methods are not suitable for current log anomaly detection needs. Automated log anomaly detection models typically rely on log template classification and use deep learning methods based on LSTM models to extract features from sequence data. However, these methods often overlook the topic characteristics of log templates. To address this issue, a log anomaly detection scheme called TALPAD is proposed, which enhances log template topic information by extracting log template topic information based on the BTM model, concatenating log template topic features and original log template features, and inputting them into a parallel LSTM model for log anomaly detection. The effectiveness of the TALPAD model is validated by using real-world Windows log datasets, and experimental results show that this method outperforms other comparative methods, achieving the best precision, recall, and F1 scores of 91.2%, 92.5%, and 91.8%, respectively.