Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts
Ben Zhi Zhou, Kyle Richardson, Xiaodong Yu, Dan Roth · 2022
Explicit decomposition modeling, which involves breaking down complex tasks into more straightforward and often more interpretable sub-tasks, has long been a central theme in developing robust and interpretable NLU systems.However, despite the many datasets and resources built as part of this effort, the majority have small-scale annotations and limited scope, which is insufficient to solve general decomposition tasks.In this paper, we look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts, particularly large-scale parallel news.We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible.For example, on semantic parsing, our model, DECOMPT5, improves 20% to 30% on two datasets, Overnight and TORQUE, over the baseline language model.We further use DECOMPT5 to build a novel decompositionbased QA system named DECOMPENTAIL, improving over state-of-the-art models, including GPT-3, on both HotpotQA and StrategyQA by 8% and 4%, respectively.