Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer
Ahmet Berk Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, Sebastian Ruder · 2022
Massively multilingual models are promising for transfer learning across tasks and languages.However, existing methods are unable to fully leverage training data when it is available in different task-language combinations.To exploit such heterogeneous supervision, we propose Hyper-X, a single hypernetwork that unifies multi-task and multilingual learning with efficient adaptation.This model generates weights for adapter modules conditioned on both tasks and language embeddings.By learning to combine task and language-specific knowledge, our model enables zero-shot transfer for unseen languages and task-language combinations.Our experiments on a diverse set of languages demonstrate that Hyper-X achieves the best or competitive gain when a mixture of multiple resources is available, while being on par with strong baselines in the standard scenario.Hyper-X is also considerably more efficient in terms of parameters and resources compared to methods that train separate adapters.Finally, Hyper-X consistently produces strong results in few-shot scenarios for new languages, showing the versatility of our approach beyond zero-shot transfer.1 NER en Pre-trained Model Pre-trained Model Fine-tuned Model ar tr Fine-tuned Model ar tr POS en Single-Task Pre-trained Model Fine-tuned Model ar tr ar tr NER POS en en Multi-Task Pre-trained Model Fine-tuned Model tr NER POS ar tr NER ar POS en en Mixed-Language Multi-Task Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš.2020a.From Zero to Hero: On the Limitations of Zero-Shot