Non-Deterministic and Semi-Supervised Log-based Anomaly Detection
Swaraj Kumar, Vishal Murgai, Zarka Bashir, Sukhdeep Singh · 2023
Log analytics is emerging as a crucial discipline for organizations aiming to derive meaningful insights and value from the vast amount of log data generated by complex applications, including traditional RAN and 5G V-RAN systems. Deep learning (DL) algorithms have the potential to enhance the effectiveness of anomaly detection approaches by automatically recognizing intricate patterns and relationships within the log data. However, the majority of these systems face challenges when dealing with dynamic and unstable log data. Furthermore, these DL algorithms often heavily rely on manually labeled training data, which can limit their ability to detect anomalies that deviate from predefined log patterns. Specifically, these models often perform sub-optimally to detect unusual log metric value patterns. In this paper, we propose non-deterministic Semi-supervised Log-Based Anomaly Detection (SSLAD) technique that comprehensively analyzes all aspects of log messages, including log parameter values, to effectively identify abnormalities arising from abnormal parameter patterns. SSLAD obviates the need for tedious manual labeling efforts by using un-supervised and leveraging bi-directional Long Short-Term Memory (bi-LSTM) models, which are known for their resilience to noisy and evolving log data. By incorporating historical anomaly knowledge, our idea enhances the accuracy and robustness of anomaly detection.