Log Data Real Time Analysis Using Big Data Analytic Framework with Storm and Hadoop
Jiake Lv, Li Yang, Xuan Wang · MATEC Web of Conferences · 2018
The log data real-time processing platform which is built using Storm On YARN integrated MapReduce and Storm that use MapReduce to complete large-scale off-line data global knowledge extraction, sudden knowledge extraction of small-scale data in Kafka buffers through Storm, and continuous real-time calculation of streaming data in combination with global knowledge. We tested our technique with the well-known KDD99 CUP data set. The experimentation results prove the system to be effective and efficient.