Control Prefixes for Parameter-Efficient Text Generation
Jordan Clive, Kris Cao, Marek Rei · 2022
Prefix-tuning is a parameter-efficient and powerful technique for adapting a pre-trained language model to a downstream application.However, it uses the same dataset-level tuned set of parameters for all examples in the dataset.We extend the framework with a dynamic method, CONTROL PREFIXES, which allows for the effective inclusion of input-dependent information, thereby demonstrating how prefixtuning can be used for controlled text generation tasks.The method incorporates attributelevel learnable representations into different layers of a pre-trained Transformer, enabling the generated text to be guided in a particular direction.We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG).Using only 0.1-2% additional trainable parameters, we show CON-TROL PREFIXES can even outperform full finetuning methods, and present state-of-the-art results on several data-to-text datasets, including WebNLG.We also examine the common case where input-dependent information is unavailable at test time and show CONTROL PREFIXES can excel in this setting also.