QuickLogS: A Quick Log Parsing Algorithm based on Template Similarity
Luyue Fang, Xiaoqiang Di, Xu Liu, Yiping Qin, Weiwu Ren, Qiang Ding · 2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · 2021
Logs are widely used in network security and management because they record runtime details in IT systems. It is difficult to gain insights from raw unstructured logs, so many researches first parse raw logs into structured templates. However, as the volume of logs grows rapidly, efficiency becomes a major concern in log parsing. In this paper, we propose a quick log parsing algorithm QuickLogS based on template similarity. QuickLogS utilizes regular expressions to replace the variables with wildcard and filters the reduplicate data to parse huge volume of unstructured logs into finite structured templates. To improve parsing efficiency, SimHash algorithm and Hamming distance are used to merge the similar templates of the same length. To the best of our knowledge, we are the first to apply the SimHash algorithm to log parsing. Besides, different with other work, we also merge the similar templates of different lengths based on the cosine similarity algorithm, which contributes to improve the parsing accuracy. QuickLogS is evaluated on six real public log datasets, and compared with four state-of-the-art log parsing algorithms. The experimental results show that QuicklogS outperforms the other parsers in terms of efficiency and accuracy.