Log Anomaly Detection Based on Bi-LSTM Feature Extraction
Yue Zhang, Dong Zhang, Feng Guo, Xiaotong Wang, Yihai Duan, Xiaonan Zhang · 2022 5th International Conference on Data Science and Information Technology (DSIT) · 2022
With the development of the Internet and the digital age continues to advance, traditional operation and maintenance methods can no longer meet the maintenance of large-scale distributed systems, AIOps emerges as the times require. In AIOps, finding anomalies is a top priority. PCA (Principal Component Analysis) is a traditional machine learning algorithm, which is used for log anomaly detection using the dimensionality reduction feature of PCA algorithm, the advantages of this detection method are: fast detection and simple model. However, the anomaly detection effect is not satisfactory, because when extracting the features of log data, the sequence features and semantic features cannot be sufficiently mined, resulting in a lower accuracy rate. For this problem, we propose a feature extraction improvement model. This model is based on Bi-LSTM (Bidirectional Long Short Term Memory) deep network to improve the feature extraction method of PCA model, extracting semantic features of log sequences by exploiting the ability of long short-term memory network to grasp contextual information, and feed the vector representation matrix containing the semantic features into the PCA anomaly detection network for anomaly detection. The model is validated on HDFS open source dataset and Inspur server dataset, and the results show that the F1-measure of this model is 36.3 % higher than that of the traditional PCA model, and 1.1 % higher than that of the single Bi-LSTM model.