Survey on Real-time Anomaly Detection Technology for Big Data Streams
Luo Yuanyan, Xuehui Du, Yi Sun · 2018
With the rapid development of cloud computing, internet of things and smart cities, a large number of related programs generate big data streams during running. These big data streams will be attacked by malicious code entrainment, DDOS, and illegal tampering of data contents in the network environment. How to detect this part of the abnormal data in the big data streams has become a hot spot of current research. In order to solve the shortcomings of the existing real-time anomaly detection technology of big data streams, the literature analysis method is used to demonstrate its necessity. The related concepts are briefly described and the key problems faced by real-time anomaly detection technology of big data streams are summarized. Through systematic research on existing typical algorithms, the algorithms are summarized into three categories: based on statistics, based on clustering and based on distance. Focus on the current latest algorithm schemes, the schemes are compared in terms of time complexity and memory consumption. And the data stream generator is used to implement each scheme on the MOA (Massive Online Analysis) platform to carry out experimental testing and data analysis of the algorithms. Finally, the current hot issues and development prospects in this field are summarized, which will provide reference for further research.