A Scalable Hate Speech Detection System for Vietnamese Social Media using Real-time Big Data Processing and Distributed Deep Learning
Van-Co Dinh, Tran-Dai Vo, Mai-Phuong T. Nguyen, Trong-Hop Do · 2023
In this study, a system to detect hate and offensive social network comments in real-time using big data and distributed deep learning technology is presented. In the offline phase, state-of-the-art deep learning models are trained in a distributed manner using to BigDL library. The trained models are then integrated into the real-time big data processing component powered by Apache Spark, which a big data framework capable of processing a huge amount of comments in real-time. In the online phase, continuous stream of comments from Facebook are crawled and channeled through Kafka to this real-time big data processing component to output hate speech detection results. These results are then are then analyzed, and the statistical data is displayed in a web-app powered by Flask. Therefore, this work not only focuses on the accuracy but also emphasizes the system’s practicality. Thanks to state-of-the-art deep learning models, the system can achieve high accuracy in the hate speech detection. With the deployed big data technology, the system can collect and process huge amounts of Facebook comments and produce statistical results in real-time.