OptimizeLog: Log Anomaly Detection and Localization based on Optimized Log Parsing in Distributed Systems
Yanjie Sun, Yali Gao, Xiaoyong Li · 2023
Modern distributed systems generate interleaved logs when performing parallel operations, and these logs become an important basis for anomaly detection and localization. To achieve more robust and accurate log anomaly detection and localization, we propose a comprehensive model, namely OptimizeLog. We built a dictionary- and length-based log parser that works stably in most log systems without parameter tuning. We significantly improve anomaly detection accuracy by introducing ALBERT-based semantic embeddings, count embeddings, and time embeddings, combined with an attentionbased Bi-GRU model. By constructing component instance forest and tree-based depth-first traversal algorithm, abnormal location in distributed systems is realized. Experimental results show that our log parsing method is 4% more accurate than other log parsers. Compared with other advanced methods such as DeepLog, LogAnomaly, and LogRobust, OptimizeLog improves the accuracy of anomaly detection by 5%, while enabling instancelevel anomaly localization on real datasets.