Benchmarking Long-tail Generalization with Likelihood Splits
Ameya Godbole, Robin Jia · 2023
In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances.We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets.We create 'Likelihood Splits' where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set.This simple approach can be customized to construct meaningful traintest splits for a wide range of tasks.Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on SPIDER, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BOOLQ, on our splits compared with the corresponding random splits.Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.