Automatic Anomaly Detection Using Unlabelled Log Data
Yujia Zhu, Geyong Min, Yulei Wu, Haozhe Wang · 2023
In this paper, we embark on a meticulous investigation of anomaly detection within OpenStack system logs, employing a two-pronged approach that ingeniously combines a task splitting method for log grouping with a two-layer stacked Long Short-Term Memory (LSTM) network for the anomaly detection phase. Our exploration commences with the extraction of typical normal operation patterns from system logs, followed by the application of our LSTM-based anomaly detection model trained on these normal patterns. Our study reveals the remarkable potential of this combined methodology in effectively distinguishing between normal and abnormal tasks, thereby offering profound insights into system operation and potential areas of concern. This exploration stands as a proof to the immense potential of modern machine learning techniques in navigating the ever-increasing complexity and volume of log data.