Large Language Models guided Generative Prompt for Dialogue Generation
Sijie Liu, Yiquan Fang, Hua Nong Cheng, Yiming Pan, Yufei Liu, Caiting Gao · 2023
The applications of large language models (LLMs) such as ChatGPT exhibit impressive comprehension and generative capabilities in dialogue task. LLMs require massive high-quality data and computational cost, which limits their application to low-resource tasks. Dialogue generation when using smaller language models like GPT-2 encounters difficulties in maintaining context consistency. To address the problem of dialogue generation under resource constraints, we propose an LLM-guided Generative Prompt method (LGP). LGP enhances the relevance and coherence of generated dialogues through a smaller model GPT-2 and generative prompt (GP). GP is produced by the proposed Prompt Network, which leverages prompt encoder to learn dialogue history features and utilizes LSTM to extract contextual temporal features. Therefore, GP shown as the simple fixed-length learnable embeddings can replace the original complex and redundant context in GPT-2. The few-shot training of GP is guided by the LLM’s responses, which facilitates GPT-2 in generating more contextually consistent and comprehensive responses. Experiments on the DailyDialog and MultiWOZ datasets show that LGP achieves high improvements in BLEU, NIST, METEOR and ROUGE-L metrics. Remarkably, LGP achieves these results with approximately 18% of the training data, surpassing other full-data-finetuning methods in automatic evaluation metrics.