Controllable Text Generation Based on Enhanced Non-Residual Attention
Huiting Hu, Xing Wang, Guohua Zhu · 2024
The typical construction method of prompts in CLM results in the combination of prompt text information and input text information being too long, and designing a prompt is challenging. Fixed template combinations can impact the grammatical structure, complicate the understanding of the prompt model, and influence the control effect of the generative model. By enhancing Non-Residual Attention, the prompt model processes prompt information to generate improved prompts. This enables the generative model to access comprehensive enhanced prompt information and input text information at any time step. At the same time, a copying mechanism is incorporated into the generative model to tackle the consistency issue of contextual text and enhance the controllability of the generative model. This enables the model output to encompass input text information or prompt text information, thereby reflecting contextual consistency and improving output control. Based on the ROCStory dataset with labeled characters, emotions, and actions, the results indicate that the enhanced Non-Residual Attention model has better control and generation effects compared to the original method.