On Measuring the Intrinsic Few-Shot Hardness of Datasets
Xinran Zhao, Shikhar Murty, Christopher D. Manning · 2022
While advances in pre-training have led to dramatic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in datasets.In particular, given a new dataset and a pre-trained model, what properties of the dataset make it few-shot learnable and are these properties independent of the specific adaptation techniques used?We consider an extensive set of recent few-shot learning methods, and show that their performance across a large number of datasets is highly correlated, showing that few-shot hardness may be intrinsic to datasets, for a given pre-trained model.To estimate intrinsic few-shot hardness, we then propose a simple and lightweight metric called Spread that captures the intuition that fewshot learning is made possible by exploiting feature-space invariances between training and test samples.Our metric better accounts for few-shot hardness compared to existing notions of hardness, and is ~8-100x faster to compute. ⋆ Equal ContributionMethod D1 D2 LMBFF 45.3 -0.4 NullPrompts 43.0 -5.7 BitFit 46.3 -3.5 AdaPET 44.9 -0.3 P-Tuning 46.3 0.3 Few-shot (Avg) 45.2 -2 Full Fine-tuning 45.3 35