EXPLORING THE BERT CROSS-LINGUAL TRANSFER FOR READING COMPREHENSION
Vasily Konovalov, P. A. Gulyaev, Alexey Sorokin, Yuri Kuratov, Mikhail Burtsev · Computational Linguistics and Intellectual Technologies · 2020
Multilingual BERT has been shown to generalize well in a zero-shot crosslingual setting. This generalization was measured on POS and NER tasks. We explore the multilingual BERT cross-language transferability on the reading comprehension task. We compare different modes of training of question-answering model for a non-English language using both English and language-specific data. We demonstrate that the model based on multilingual BERT is slightly behind the monolingual BERT-based on Russian data, however, it achieves comparable results with the language-specific variant on Chinese. We also show that training jointly on English data and additional 10,000 monolingual samples allows it to reach the performance comparable to the one trained on monolingual data only.