Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels
Nicholas Pangakis, Sam Wolken · 2024
Computational social science (CSS) practitioners often rely on human-labeled data to finetune supervised text classifiers.We assess the potential for researchers to augment or replace human-generated training data with surrogate training labels from generative large language models (LLMs).We introduce a recommended workflow and test this LLM application by replicating 14 classification tasks and measuring performance.We employ a novel corpus of English-language text classification data sets from recent CSS articles in high-impact journals.Because these data sets are stored in password-protected archives, our analyses are less prone to issues of contamination.For each task, we compare supervised classifiers finetuned using GPT-4 labels against classifiers fine-tuned with human annotations and against labels from GPT-4 and Mistral-7B with fewshot in-context learning.Our findings indicate that supervised classification models fine-tuned on LLM-generated labels perform comparably to models fine-tuned with labels from human annotators.Fine-tuning models using LLMgenerated labels can be a fast, efficient and cost-effective method of building supervised text classifiers.