Active learning for e-rulemaking: public comment categorization
Stephen Purpura, Claire Cardie, Jesse Simons · Scholarship @ Cornell Law (Cornell University) · 2008
We address the e-rulemaking problem of reducing the man-ual labor required to analyze public comment sets. In cur-rent and previous work, for example, text categorization techniques have been used to speed up the comment analysis phase of e-rulemaking — by classifying sentences automati-cally, according to the rule-specific issues [2] or general top-ics that they address[7, 8]. Manually annotated data, how-ever, is still required to train the supervised inductive learn-ing algorithms that perform the categorization. This pa-per, therefore, investigates the application of active learning methods for public comment categorization: we develop two new, general-purpose, active learning techniques to selec-tively sample from the available training data for human la-beling when building the sentence-level classifiers employed in public comment categorization. Using an e-rulemaking corpus developed for our purposes [2], we compare our meth-ods to the well-known query by committee (QBC) active learning algorithm [5] and to a baseline that randomly se-lects instances for labeling in each round of active learning. We show that our methods statistically significantly exceed the performance of the random selection active learner and the query by committee (QBC) variation, requiring many fewer training examples to reach the same levels of accuracy on a held-out test set. This provides promising evidence that automated text categorization methods might be used effectively to support public comment analysis.