TCN-Log2Vec: A Comprehensive Log Anomaly Detection Framework based on Optimized Log Parsing and Temporal Convolutional Network
Zhe Jiang, Yali Gao, Jie Yuan, Kaiguo Yuan, Xiaoyong Li · 2023
Logs that record execution information and system status, are useful for anomaly detection. Traditionally, developers manually checked logs through keyword search or rule matching. However, as the number of logs increases, this approach becomes unrealistic. More and more methods of log anomaly detection based on machine learning and deep learning have been proposed. In order to detect log anomalies accurately and effectively, this paper proposes a comprehensive framework, namely TCN-Log2Vec. Based on Drain, a prevalent and effective method to extract the log templates from the original logs, TCN-Log2Vec has done some optimization work in log parsing. It also captures sequence information, quantitative information, and semantic information from the original log. Meanwhile, it designs the anomaly detection module based on TCN (Temporal Convolutional Network) to achieve parallel processing. Our log parsing method achieves 99% accuracy on HDFS datasets and 94% accuracy on BGL datasets. We compare our anomaly detection framework with other advanced methods including DeepLog, LogAnomaly, and LogRobust, the experiment results show that our model has better performance.