LGLog: Semi-supervised Graph Representation Learning for Anomaly Detection based on System Logs
Jialong Liu, Yanni Tang, Jiamou Liu, Kaiqi Zhao, Wu Chen · 2023
Anomaly detection is an important task that improves the maturity and stability of a software during its development. System logs record rich information about the running states of the software and reveal key insights of anomalous behaviors. This paper addresses anomaly detection using system log data and aims to resolve two challenges: First, different from most existing supervised learning-based anomaly detection methods that rely heavily on expensive, manually-curated labels, we aim to design an algorithm to make the most of scarce label information. Second, as a typical software system would contain very few anomalies, we aim to address the data imbalance issue which is often overlooked by existing studies. To address the challenges above, we propose LGLog, a semi-supervised anomaly detection framework that is based on system logs. First, LGLog transforms log sequences into graphs and employs an unsupervised graph learning model for pre-training. Then, LGLog mitigates the data imbalance issue by learning significant latent space representation of log events via reconstruction loss and node invariance loss, and further applies a weight balance method. Experiments indicate that LGLog outperforms compared approaches, and demonstrates the effectiveness of LGLog in the presence of scarce labels and imbalanced log data.