Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models
Jiaxin Zhang, Wendi Cui, Yiran Huang, Kamalika Das, Sricharan Kumar · 2024
Large language models (LLMs) are proficient in capturing factual knowledge across various domains.However, refining their capabilities on previously seen knowledge or integrating new knowledge from external sources remains a significant challenge.In this work, we propose a novel synthetic knowledge ingestion method called Ski, which leverages fine-grained synthesis, interleaved generation, and assemble augmentation strategies to construct high-quality data representations from raw knowledge sources.We then integrate Ski and its variations with three knowledge injection techniques: Retrieval Augmented Generation (RAG), Supervised Fine-tuning (SFT), and Continual Pre-training (CPT) to inject and refine knowledge in language models.Extensive empirical experiments are conducted on various question-answering tasks spanning finance, biomedicine, and open-generation domains to demonstrate that Ski significantly outperforms baseline methods by facilitating effective knowledge injection.We believe that our work is an important step towards enhancing the factual accuracy of LLM outputs by refining knowledge representation and injection capabilities.1Raw knowledge context: "The annual contribution limit for a health savings account (HSA) in 2024 is $4,150 for individuals with self-only coverage and $8,300 for individuals with family coverage.These limits are about a 7% increase from 2023.Individuals who are 55 or older can contribute an additional $1,000, bringing the total to $4,850 for individuals and $8,750 for families…" Generate hypothetical questions given ngram knowledge contexts -"Holistic Detail-Oriented"• 1-gram Q -"What is the HSA contribution limit for individuals with self-only coverage in 2024" • 2-gram Q -"What is the HSA contribution limit for 2023?" • 3-gram Q -"How much more can individuals contribute if aged over 55 than younger?" Fine-grained SynthesisGenerate question and answer simultaneously given knowledge -"Aligned Harmony"• Context: The annual contribution limit for a health savings account (HSA) in 2024 is $4,150 for individuals with selfonly coverage and $8,300 for … • Question: What is the HSA contribution limit for individuals with self-only coverage in 2024?• Answer: $4150 Interleaved GenerationAssemble n-gram synthesis and question /answer/context pairs -"Repetition with diversity"• Assembles of question, answer, and context pairs: