CFTL: System Log Parsing Method Driven from Clustering According to First Token and Length for Anomaly Detection
Jie Hu, Lingchun Long, He Sui, Zhaojun Gu, Guangming Zheng · Applied Sciences · 2025
Logs are extensively used to analyze the running states of systems in many fields, such as cyber security, performance monitoring, and fault diagnosis, by recording events representing real-time system states. Conventional system log parsing methods are time-consuming and are prone to overfitting or underfitting. Thus, automatic parsing methods, known as log parsers, have been proposed. However, most log parsers do not exhibit both high accuracy and low running time. In addition, they typically overfit during log template generation; thus, log parsers cannot effectively be deployed in large-scale distributed systems. To solve these problems, this study proposes an efficient heuristic log parsing method. The proposed method first clusters log messages by the first token and their length, then uses specific separation rules to divide them into refined groups, and finally matches the corresponding log templates. The performance of the proposed method was evaluated using reliable datasets and tools. The experimental results demonstrate that the proposed method not only exhibits high accuracy but also a low running time. The number of log events parsed by log parsers of log templates generated by the proposed method is close to the real count, and the proposed method also exhibits good accuracy for subsequent anomaly detection tasks. In addition, the proposed method is lightweight because it employs the most valuable parsing rules. Thus, the proposed method is suitable for deployment in large-scale distributed systems.