Comparative Study of Leveraging Big Data Processing Techniques for Sentiment Analysis
Chris-Ern-Zer Wong, Lee-Yeng Ong, Meng-Chew Leow · 2023
Sentiment analysis, an essential task in natural language processing, plays a pivotal role in understanding sentiment and opinions expressed in textual data. However, with the exponential growth of social media and online platforms, the sheer volume of textual data presents challenges for efficient processing. Traditional approaches struggle to cope with the increased data size, necessitating the adoption of big data processing techniques. This study presents a comparative performance analysis of sentiment analysis, evaluating the utilization of a big data processing framework. The study compares three machine learning algorithms for sentiment analysis with and without the implementation of big data processing techniques, focusing on model training efficiency. Additionally, two textual feature extraction techniques are examined to assess their impact on the results. Evaluation of the models' performance is based on the average execution time for training. The study's findings indicate that SparkML's Random Forest significantly outperforms the traditional sci-kit learn's Random Forest in terms of training time.