Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering
Mingda Chen, Xilun Chen, Wen-tau Yih · 2024
Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models (LLMs).While powerful, these LLMs usually contain tens or hundreds of billions of parameters, making them rather inefficient at inference time.To improve performance of smaller language models, we propose a data synthesis framework for multi-hop question answering that requires less than 10 humanannotated question answer pairs.Our framework depends only on rich, naturally-occurring relationships among documents and is built upon the data generation functions parameterized by LLMs and prompts.We synthesize millions of multi-hop questions and claims to finetune language models, evaluated on popular benchmarks for multi-hop question answering and fact verification.Empirically, our approach improves model performance significantly, allowing the finetuned models to be competitive with GPT-3.5 based approaches while being almost one-third the size in parameter count.What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?Query1: the eastern section of the Colorado orogeny Query2: the elevation range for the High Plains Unanswerable Question generation Query generation Query verification Question: What is the elevation range for … Query: the eastern section of the Colorado … Observation: 1: The Colorado orogeny, or Colorado orogen, was an orogeny … … Answer: 1,800 to 7,000 ft Question answering 1,800 to 7,000 ft Randomly sample document pairsThe causes of World War II are debated, but contributing factors included the Second Italo-Ethiopian War, Spanish Civil War, Second Sino-Japanese War, Soviet-Japanese border conflicts, the rise of fascism in Europe, and European tensions in the aftermath of World War I.The Second Italo-Ethiopian War, also referred to as the Second Italo-Abyssinian War, was a war of aggression which was fought between Italy and Ethiopia from October 1935 to February 1937. Events occurred in sequenceThe Colorado orogeny, or Colorado orogen, was an orogeny in Colorado and surrounding areas which was a part of the development of the ancestral Rockies.The eastern sector extends into the High Plains and is called the Central Plains orogeny.The High Plains are a subregion of the Great Plains.From east to west, the High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130 m). Extra geographical informationNew York, often called New York City or NYC, is the most populous city in the United States.With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km 2 ), the city is the most densely populated major city in the United States.NYC is more than twice as populous as Los Angeles, the nation's second-largest city.Los Angeles, often referred to by its initials L.A., officially the City of Los Angeles, is the most populous city in the U.S. state of California. Extra demographic information