A Scalable Real-Time Burst Detection System based on Kleinberg Algorithm
Min Zhou Luo, Akihiro Yamanaka, Masato Sawada · 人工知能学会全国大会論文集 · 2014
Real-time burst detection is an important topic in data stream analysis. An effective way to find burst events is using a sliding time-window to record the frequency of evens that happened in each slide interval. Although many burst detection methods have been conducted based on this idea, only a handful of them are suitable for real-time detection. In this paper, we propose a real-time burst detection system based on Kleinberg algorithm and Jubatus. It is able to detect 100 events’ burst whose arriving frequency is of several milliseconds. By deploying our proposal on Jubatus framework, 10x times larger numbers of events could be detected at the same accuracy with the same regular specs of PCs. According to our knowledge, this is the first system for scalable real-time burst detection. Experimental evaluations of our prototype system on tweets and financial data analysis demonstrate the feasibility and scalability of our proposal.