K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
Song Xu, Haoran Li, Peng Yuan, Yujia Wang, Youzheng Wu, Xiaodong He, Ying Liu, Bowen Zhou · 2021
Existing pre-trained language models (PLMs) have demonstrated the effectiveness of selfsupervised learning for a broad range of natural language processing (NLP) tasks.However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios.In this paper, we propose K-PLUG, a knowledgeinjected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks.We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge.Specifically, we propose five knowledgeaware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domainspecific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities.K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multiturn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks.Our code is available at https:// github.com/xu-song/k-plug.