Research on Dynamic Expansion Method of Multi Source Log Index
Feng Zou, Peng Lu, Guo Wu · 2023
To address the difficulties of index expansion caused by the real-time nature and multi-source heterogeneity of logs in network security analysis, a dynamic index expansion method for multi-source heterogeneous logs was studied. The strategy based on rolling windows achieves dynamic expansion of real-time log index capacity, and ETL for multi-source logs is implemented based on reflection mechanism. The experimental results show that the proposed method can achieve a write performance of 450,000 logs per second in a small-scale Elasticsearch cluster, and can delete aging historical logs more than 100 times faster than traditional methods, demonstrating the effectiveness of the method.