Microblog Sentiment Classification Using Parallel SVM in Apache Spark
Bo Yan, Zijiang Yang, Yitian Ren, Xing Tan, Eric Liu · 2017
In the information age, sentiment classification of Internet topics is of great significance. This paper proposes a microblog sentiment classification approach with parallel support vector machine (SVM). The proposed method integrates the features of microblog with preprocessing to ensure the data suitable for sentiment classification. After the preprocessing process, Apache Spark parallel SVM is used to execute the classification. SVM is one of the most popular algorithms in text classification. It fits small scale and nonlinear problems. However, SVM takes very long when dealing with big data. We apply Spark to parallelize SVM with Radial Basis Function (RBF) kernel function. The introduction of Apache Spark results in outstanding performance in machine learning compared to Hadoop. The experiments show that Spark increases the execution speed of SVM significantly. At the same time the classification accuracy is also increased by information gain (IG) approach in the preprocessing and kernel function parameter selection.