Dataset Distillation with Attention Labels for Fine-tuning BERT
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, Manabu Okumura · 2023
Dataset distillation aims to create a small dataset of informative synthetic samples to rapidly train neural networks that retain the performance of the original dataset.In this paper, we focus on constructing distilled fewshot datasets for natural language processing (NLP) tasks to fine-tune pre-trained transformers.Specifically, we propose to introduce attention labels, which can efficiently distill the knowledge from the original dataset and transfer it to the transformer models via attention probabilities.We evaluated our dataset distillation methods in four various NLP tasks and demonstrated that it is possible to create distilled few-shot datasets with the attention labels, yielding impressive performances for finetuning BERT.Specifically, in AGNews, a fourclass news classification task, our distilled fewshot dataset achieved up to 93.2% accuracy, which is 98.5% performance of the original dataset even with only one sample per class and only one gradient step.