ParsTwiNER: A Corpus for Named Entity Recognition at Informal Persian
MohammadMahdi Aghajani, AliAkbar Badri, Hamid Beigy · 2021
As a result of unstructured sentences and some misspellings and errors, finding named entities in a noisy environment such as social media takes much more effort.ParsTwiNER contains about 250k tokens, based on standard instructions like MUC-6 or CoNLL 2003, gathered from Persian Twitter.Using Cohen's Kappa coefficient, the consistency of annotators is 0.95, a high score.In this study, we demonstrate that some state-of-the-art models degrade on these corpora, and trained a new model using parallel transfer learning based on the BERT architecture.Experimental results show that the model works well in informal Persian as well as in formal Persian.