Research and Implementation of Log-Based Anomaly Detection Platform Based on ELK+Kafka
Shuai Xu, Ziyang Meng, Han Wang · 2024
Anomaly detection is vitally essential to modern large-scale distributed systems. Over the years,many log-based anomaly detection methods have been proposed. Logs record the real-time running information of processes in distributed systems and show the running status of the processes. However, when the reliability issues that happen in distributed systems, operators often inspect the logs manually with multi-rule matching and multi-keyword search. Additionally, The increasing scale and complexity of distributed systems, make the volume of logs explode, which renders the infeasibility of manual inspection. To reduce manual effort, we provide a log-based anomaly detection platform, which is based on ELK (Elasticsearch, Logstash, Kibana), Kafka and deep learning-based methods. Our proposed platform is evaluated using two different publicly-available production log datasets, and our results demonstrate that it achieves the$F_{1}$-score of 0.792 and 0.829, respectively, outperforming the manually inspecting log anomalies.