Twitter-scale New Event Detection via K-term Hashing
Dominik Wurzer, Victor P. Lavrenko, Miles Osborne · 2015
First Story Detection is hard because the most accurate systems become progressively slower with each document processed.We present a novel approach to FSD, which operates in constant time/space and scales to very high volume streams.We show that when computing novelty over a large dataset of tweets, our method performs 192 times faster than a state-of-the-art baseline without sacrificing accuracy.Our method is capable of performing FSD on the full Twitter stream on a single core of modest hardware.