Czert – Czech BERT-like Model for Language Representation
Jakub Sido, Ondřej Pražák, Pavel Přibáň, Jan Pašek, Michal Seják, Miloslav Konopík · 2021
This paper describes the training process of the first Czech monolingual language representation models based on BERT and ALBERT architectures.We pre-train our models on more than 340K of sentences, which is 50 times more than multilingual models that include Czech data.We outperform the multilingual models on 9 out of 11 datasets.In addition, we establish the new state-of-the-art results on nine datasets.At the end, we discuss properties of monolingual and multilingual models based upon our results.We publish all the pretrained and fine-tuned models freely for the research community.