Design and Implementation of Log Analysis System Based on ELK and Spark Streaming
Csmc Technologies · Electronic Science and Technology · 2015
In the era of big data,real-time monitoring of running status and operation analysis is significant to the data platform components.The realization of the objective needs log processing solutions in accordance with company's own application's situation.This article is based on the popular open source log process solution ELK Stack,discusses the scheme of the main functions of the components and working principle,and the methods of processing the database log,and integrated with Kafka to ensure the data reliability,and also integrated with Spark Streaming in order to realize the real-time alarm.This paper provides a reference for data platform operational monitoring environment.