Research on Anomaly Detection System of Online Multi-node Log Flow
ZHAO Yining WANG Xiaodong · DOAJ (DOAJ: Directory of Open Access Journals) · 2020
With the increasing amount of logs produced by nodes in CNGrid, traditional manual methods for abnormal log analysis can no longer meet the need of daily analysis. In order to analyze the log automatically and efficiently, a two-stage detection method is proposed in this paper. In the first stage, the log patterns are classified during preprocessing, then the principal component analysis is used for anomaly detection and the sequence of log types is defined as a log flow pattern. The abnormal flow patterns obtained from anomaly detection are extracted by the definition. Finally, the hierarchical clustering algorithm is used to simplify the results of the flow pattern and the results are saved. In the second stage, through the detection model and flow pattern obtained in the first stage, the log flow information can be monitored and analyzed in real time and the corresponding flow pattern can be matched. Finally, the experiment is carried out on real logs in CNGrid, and the results are visualized in real time. These greatly reduce the manual work of operations.