Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
Raymond Li, Gabriel Murray, Giuseppe Carenini · 2023
In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient finetuning (PEFT) setting.In our approach, parallel adapter modules encoding different linguistic structures are combined using a novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates are used to determine the importance of these modules at each layer of the model.To reduce the number of parameters, we first train the model for a fixed small number of steps before pruning the experts based on their importance scores.Our experiment results with three different pretrained models show that our approach can outperform state-of-the-art PEFT methods with a comparable number of parameters.In addition, we provide additional analysis to examine the experts selected by each model at each layer to provide insights for future studies.