A log parsing method based on word attributes and parsing trees
Haiyang Liu, Yuan Feng · 2024
With the rapid development of science and technology, system security is facing an increasing number of threats, among which system vulnerabilities have brought serious challenges to information security. In such an environment, system logs, as key data that record the operating status of computer systems, user operations and various events, play an extremely important role in monitoring system health and quickly locating and resolving system failures. However, the current accuracy of log parsing is not high enough to meet the actual demand; the parsing process is inefficient and difficult to adapt to the processing requirements of large data log files. Aiming at the problems of insufficient accuracy of log parsing and low efficiency of parsing process, a log parsing method AT-parser (Log parser based on word attributes and parse trees) is proposed. Through the log preprocessing, the log content used in the subsequent construction of log templates can be separated, the log content combined with word attribute filtering, retaining the key words, constructing a string set through the key words, and then performing similarity clustering based on the string set to classify the log strings into different log groups. Finally, the final template sequence is generated by constructing a parse tree, which improves the accuracy and speeds up the parsing efficiency. Finally, the effectiveness of AT-Parser is verified by experiments on public datasets.