A real-time knowledge extracting system from social big data using distributed architecture
Youngsub Han, Hyeoncheol Lee, Yanggon Kim · 2015
A huge amount of data is being generated by social media in real time. Accordingly, demands for extracting meaningful information from the social data have been dramatically increased. However, most of the previous research encompasses potential problems with data processing, management and analysis in real time. In this paper, we propose a distributed system architecture for generating meaningful information from text-based social data. The system collects data from multi-source channels, such as Twitter, YouTube, and The New York Times. Also, the system extracts terms and sentiment from each document using data mining technologies. In addition, the system uses HDFS, Map-reduce, and message service to handle the huge data. By analyzing keywords in texts and user account information, the system generates a summary of results including terms, sentiments and data variations for further analysis, including reputation, social trends, and customer reactions. The experiment results show that our approach is able to effectively process the social data in real time.