Optimizing log parsing efficiency: a heuristic-based distributed system approach
Dongqing Liu · 2024
System logs are widely used in system management for reliability assurance and serve as crucial data sources for detecting system anomalies. Prior to anomaly detection, it is necessary to parse raw textual logs into structured logs that can be recognized by anomaly detection models. To address the inefficiency of existing log parsing algorithms, this study proposes a distributed system log parsing task allocation model and a heuristic-based log parsing algorithm. By combining the task allocation model with the parsing algorithm, the log parsing task is completed. Experimental results demonstrate the feasibility and effectiveness of the proposed model and algorithm, which can reduce parsing time and enhance parsing efficiency. Future research could explore log parsing methods based on deep learning and log parsing algorithms integrating natural language processing techniques to improve the accuracy and applicability of log parsing.