Research on real-time log analysis system based on elastic stack and Flink
Taizhi Lv, Zhang Li, Peiyi Tang · 2023
The log data records the operation information of a software platform. The analysis of the log helps the manager to master the platform operation and ensure the normal operation of the platform. The rapid development of information technology has led to an explosive growth of log size. The traditional log processing architecture cannot meet the processing of massive logs. In order to process massive log data in real-time, a real-time log analysis system based on Elastic stack and Flink is designed and implemented. Based on Elastic stack, massive log data is acquired, transferred and stored in real-time. Based on Flink which is a streaming computing framework, log information is analyzed in realtime. Log analysis and statistical results are stored in MySQL database. Visualization is realized through front-end and back-end separation technology. The system has been deployed on a business platform. It can run stably against the daily average log of more than 10G, which proves the stability and reliability of the system.