Implementation of a Large-Scalable Social Data Analysis System Based on MapReduce
Hyeokju Lee, Joon Her, Sung‐Ryul Kim · 2011
In recent years, rapid advancement of internet and communication technologies as a way of social communications has emerged as the SNS. This way of communication needs social data analysis method for the meaningful information. In addition, rapidly increasing social data causes the lack of reliability of the information in the community. Therefore, an effective semantic analysis method is required. This paper proposed a Large-scalable Social Data Analysis System and evaluated it. The proposed system consists of two parts: social data gathering agent module and social data analysis module. The main role of gathering agent is to collect text type's data from SNS with gigabyte unit. Improved TF_IDF and Weighted-MINAX are used for semantic analysis of collected social data from the gathering agent. Thus, the improved social data analysis module supports meaningful information service which contains high priority words. We implemented the improved social data analysis module that uses Hadoop MapReduce programing model. In addition, we evaluated large-scalable social data analysis system in terms of processing time and accuracy.