mPMR: A Multilingual Pre-trained Machine Reader at Scale
Weiwen Xu, Xin Yan Li, Wai Pang Lam, Lidong Bing · 2023
We present multilingual Pre-trained Machine Reader (mPMR), a novel method for multilingual machine reading comprehension (MRC)style pre-training.mPMR aims to guide multilingual pre-trained language models (mPLMs) to perform natural language understanding (NLU) including both sequence classification and span extraction in multiple languages.To achieve cross-lingual generalization when only source-language fine-tuning data is available, existing mPLMs solely transfer NLU capability from a source language to target languages.In contrast, mPMR allows the direct inheritance of multilingual NLU capability from the MRCstyle pre-training to downstream tasks.Therefore, mPMR acquires better NLU capability for target languages.mPMR also provides a unified solver for tackling cross-lingual span extraction and sequence classification, thereby enabling the extraction of rationales to explain the sentence-pair classification process. 1