Topic Extraction Method from Millions of Tweets Based on Fast Feature Selection Technique CWC
Takako Hashimoto, Dave Shepard, Tetsuji Kuboyama, Kilho Shin · 2016
Social media offers a wealth of insight into how significant topics such as the Great East Japan Earthquake, the Arab Spring, and the Boston Bombing affect individuals. The scale of available data, however, can be intimidating: during the Great East Japan Earthquake, over 8 million tweets were sent each day from Japan alone. Conventional word vector-based topic-detection techniques for social media that use Latent Semantic Analysis, Latent Dirichlet Allocation, or graph community detection often cannot scale to such a large volume of data due to their space and time complexity. To alleviate this problem, we have already proposed an efficient method for topic extraction by leveraging our original fast feature selection algorithm, CWC, which vastly reduces the number of features to track. While we begin with word count vectors of authors and words for each time slot (in our case, every 30 minutes), we make clusters from each time slot by a matrix decomposition technique to identify clusters and adapt CWC to extract discriminative words from each cluster. This method makes it possible to detect topics from high dimensional datasets. In this paper, to demonstrate our method's effectiveness, we extract topics from a dataset of over two hundred million tweets sent following the Great East Japan Earthquake and compare them with the result extracted by LDA, the current most popular topic extraction method. With CWC, we can identify topics from this dataset with great speed and accuracy.