Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study
Aditya Siddhant, Zachary C. Lipton · 2018
Several recent papers investigate Active Learning (AL) for mitigating the datadependence of deep learning for natural language processing.However, the applicability of AL to real-world problems remains an open question.While in supervised learning, practitioners can try many different methods, evaluating each against a validation set before selecting a model, AL affords no such luxury.Over the course of one AL run, an agent annotates its dataset exhausting its labeling budget.Thus, given a new task, an active learner has no opportunity to compare models and acquisition functions.This paper provides a largescale empirical study of deep active learning, addressing multiple tasks and, for each, multiple datasets, multiple models, and a full suite of acquisition functions.We find that across all settings, Bayesian active learning by disagreement, using uncertainty estimates provided either by Dropout or Bayes-by-Backprop significantly improves over i.i.d.baselines and usually outperforms classic uncertainty sampling.