XeroAlign: Zero-shot cross-lingual transformer alignment
Milan Gritta, Ignacio Iacobacci · 2021
The introduction of transformer-based crosslingual language models brought decisive improvements to multilingual NLP tasks.However, the lack of labelled data has necessitated a variety of methods that aim to close the gap to high-resource languages.Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s).We introduce XeroAlign, a simple method for taskspecific alignment of cross-lingual pretrained transformers such as XLM-R.XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages.The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-ofthe-art zero-shot results on three multilingual natural language understanding tasks.XLM-RA performs on par with state-of-the-art models on a cross-lingual adversarial paraphrasing task and its text classification accuracy exceeds that of XLM-R trained with labelled data.