Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization
Inbar Oren, Jonathan Herzig, Jonathan Berant · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
Modern semantic parsers suffer from two principal limitations.First, training requires expensive collection of utterance-program pairs.Second, semantic parsers fail to generalize at test time to new compositions/structures that have not been observed during training.Recent research has shown that automatic generation of synthetic utterance-program pairs can alleviate the first problem, but its potential for the second has thus far been under-explored.In this work, we investigate automatic generation of synthetic utterance-program pairs for improving compositional generalization in semantic parsing.Given a small training set of annotated examples and an "infinite" pool of synthetic examples, we select a subset of synthetic examples that are structurally-diverse and use them to improve compositional generalization.We evaluate our approach on a new split of the schema2QA dataset, and show that it leads to dramatic improvements in compositional generalization as well as moderate improvements in the traditional i.i.d setup.Moreover, structurally-diverse sampling achieves these improvements with as few as 5K examples, compared to 1M examples when sampling uniformly at random -a 200x improvement in data efficiency.