Semi-supervised log anomaly detection based on clustering
Nan Lin, Yuejun Jia · 2024
As computer systems become increasingly complex, deep learning methods for rapidly analyzing and pinpointing anomalies in system logs are gaining widespread application to ensure smooth system operation. Addressing the issues of many existing methods that require substantial amounts of labeled data and insufficient utilization of temporal features in logs, we propose a semi-supervised log anomaly detection method. Initially, log templates are extracted using the Drain template parsing technique. Subsequently, BERT is employed to extract deep semantic feature vectors from logs and to derive time feature vectors. Unsupervised clustering algorithms are then used to estimate labels for unlabeled samples, tackling the problem of insufficiently annotated data in practical log anomaly detection scenarios. Finally, anomaly detection is achieved using a Attn-based Bi-LSTM model. Experimental results on two datasets, HDFS and BGL, demonstrate that our proposed method achieves notable improvements in terms of accuracy and recall, thereby validating the effectiveness of our work.