Modular and Parameter-Efficient Fine-Tuning for NLP Models
Sebastian Ruder, Jonas H. Pfeiffer, Ivan Vulić · 2022
State-of-the-art language models in NLP perform best when fine-tuned even on small datasets, but due to their increasing size, finetuning and downstream usage have become extremely compute-intensive.Being able to efficiently and effectively fine-tune the largest pretrained models is thus key in order to reap the benefits of the latest advances in NLP.In this tutorial, we provide a comprehensive overview of parameter-efficient fine-tuning methods.We highlight their similarities and differences by presenting them in a unified view.We explore the benefits and usage scenarios of a neglected property of such parameterefficient models-modularity-such as composition of modules to deal with previously unseen data conditions.We finally highlight how both properties-parameter efficiency and modularity-can be useful in the real-world setting of adapting pre-trained models to under-represented languages and domains with scarce annotated data for several downstream applications.1