LogKT: Hybrid Log Anomaly Detection Method for Cloud Data Center
Xuedong Ou, Jing Liu · 2023
Log anomaly detection is a fairly indispensable log analysis task for reliability and maintainability in cloud data center. By performing tasks such as log parsing and feature extraction on logs, which are common and valid data, a model with self-judgment capability can be trained for log anomaly detection. Improving the model used for anomaly detection is the main line of research in the current anomaly detection field. However, the data set partitioning method during anomaly detection also has an important impact on the results of anomaly detection, which should be given more considerations. Most of the existing anomaly detection models are single-architecture models, which cannot make full use of the multiple forms of information that logs have. This paper proposes a hybrid anomaly detection method, named LogKT, which is divided into two parts. First, a new dataset partitioning method is constructed based on time-series, randomness and imbalances of logs. It is a dataset partitioning method that fits the characteristics of log anomaly detection from the aspects of time-series feature preservation, sampling range expansion and training method change. Then, we further propose a hybrid anomaly detection model based on a Transformer and Bi-LSTM models, which can extract features from multiple information of logs and can fit well with the dataset partitioning method. Finally, we perform validation experiments on two public datasets, and the experimental results show that our LogKT approach has superior anomaly detection accuracy compared with baseline methods.