PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation
Yixin Wan, Kuan-Hao Huang, Kai-Wei Chang · 2023
Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures.Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process.Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost.We introduce two methods to instruct a model's encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect).In contrast to traditional finetuning methods for this task, PIP is a computeefficient alternative with 10× times less learnable parameters.Compared to existing prefixtuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.