LogTransformer: Transforming IT System Logs Into Events Using Tree-Based Approach
Yuanyuan Fu, Jian Xu · IEEE Transactions on Network and Service Management · 2024
As an important outcome of complex IT systems in operation, logs provide valuable information for system operation and maintenance. Log event (or template) extraction plays a vital role in log analysis, as its accuracy significantly impacts follow-up tasks such as log anomaly detection and event pattern discovery. Despite achieving high accuracy on specific system logs, existing log event extraction approaches still struggle with low accuracy and instability when handling logs from heterogeneous systems or logs with variable-length parameters. To address these issues, this paper proposes LogTransformer, an online event extraction approach based on a tree structure. A tree-based log content parsing approach is proposed to perform log event extraction by comparing the similarity between a log tree representing an incoming log message and an event tree representing a specific log template. Extensive experiments are conducted on sixteen benchmark log datasets to evaluate the effectiveness, robustness, and efficiency of the proposed approach. The experimental results demonstrate that an average accuracy exceeds 90%, surpassing the state-of-the-art online log parser, Drain.