A Text-to-Text Model for Multilingual Offensive Language Identification
Tharindu Ranasinghe, Marcos Zampieri · 2023
The ubiquity of offensive content on social media is a growing cause for concern among companies and government organizations.Recently, transformer-based models such as BERT, XL-NET, and XLM-R have achieved state-of-theart performance in detecting various forms of offensive content (e.g.hate speech, cyberbullying, and cyberaggression).However, the majority of these models are limited in their capabilities due to their encoder-only architecture, which restricts the number and types of labels in downstream tasks.Addressing these limitations, this study presents the first pretrained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5) trained on two large offensive language identification datasets; SOLID and CCTK.We investigate the effectiveness of combining two datasets and selecting an optimal threshold in semi-supervised instances in SOLID in the T5 retraining step.Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks.Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5 and evaluate its performance on a set of six different languages (German, Hindi, Korean, Marathi, Sinhala, and Spanish).The results demonstrate that this multilingual model achieves a new state-of-the-art on all the above datasets, showing its usefulness in multilingual scenarios.Our proposed T5-based models will be made freely available to the community.