Comparative analysis of two-class and multi-class toxicity detection using multi-source gaming chat data
Kasandika Andariefli, Jervino Leonard, Vincentius Dewanto, Andien Dwi Novika · Procedia Computer Science · 2025
Toxic chat remains a significant problem in competitive online games like Dota 2, where abusive language can degrade player experience and disrupt community health. This study focuses on detecting such messages by training deep learning models on a merged dataset from Kaggle and Hugging Face, combining chat logs labelled for toxicity. We evaluated two architectures: a Long Short-Term Memory (LSTM) model and a Convolutional Neural Network (CNN) with a 1D convolutional layer and GlobalMaxPooling. Results showed that the LSTM model struggled, predicting only the majority non-toxic class, which led to a misleading overall test accuracy of 53.4% and zero scores for the minority classes. In contrast, the CNN achieved an overall accuracy of 79.9% and demonstrated balanced performance, with F1-scores of 0.64 and 0.66 for toxic and severe-toxic classes, respectively. These results highlight CNN’s ability to capture local linguistic patterns in short chat messages, despite the dataset’s significant class imbalance. Our findings emphasize the importance of addressing data imbalance in toxicity detection and selecting model architectures suited for short, dynamic gaming chat environments. These insights can inform the development of robust moderation tools in online gaming communities, promoting healthier and more inclusive interactions. Future research will explore integrating transformer-based models and data balancing techniques to further improve detection accuracy.