MirasText: An Automatically Generated Text Corpus for Persian
Behnam Sabeti, Hossein Abedi Firouzjaee, Ali Janalizadeh Choobbasti, S. H. E. Mortazavi Najafabadi, Amir Vaheb · 2018
Natural Language Processing is one of the most important fields of artificial intelligence.The rapid growth of digital content has made this field both practical and challenging at the same time.As opposed to less-resourced languages like Persian, there are several text corpora in dominant languages like English which can be used for NLP applications.In this paper, MirasText which is an automatically generated text corpus for Persian language is presented.In this study, over 250 Persian websites were crawled and several fields like content, description, keywords, title, etc have been extracted to generate MirasText.Topic modeling and language modeling are used to validate the generated corpus.MirasText has over 2.8 million documents and over 1.4 billion tokens, which to our knowledge is the largest Persian corpus currently available.