The Dark Side of the Language: Pre-trained Transformers in the DarkNet
Leonardo Ranaldi, Aria Nourbakhsh, Arianna Patrizi, Elena Sofia Ruzzetti, Dario Onorati, Michele Mastromattei, Francesca Fallucchi, Fabio Massimo Zanzotto · 2023
Pre-trained Transformers are challenging human performances in many NLP tasks.The massive datasets used for pre-training seem to be the key to their success on existing tasks.In this paper, we explore how a range of pretrained Natural Language Understanding models perform on definitely unseen sentences provided by classification tasks over a DarkNet corpus.Surprisingly, results show that syntactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning.Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results.This suggests that huge pre-training corpora may give Transformers unexpected help since they are exposed to many of the possible sentences.