Bag of What? Simple Noun Phrase Extraction for Text Analysis
Abram Handler, Matthew James Denny, Hanna Wallach, Brendan O’Connor · 2016
Social scientists who do not have specialized natural language processing training often use a unigram bag-of-words (BOW) representation when analyzing text corpora.We offer a new phrase-based method, NPFST, for enriching a unigram BOW.NPFST uses a partof-speech tagger and a finite state transducer to extract multiword phrases to be added to a unigram BOW.We compare NPFST to both ngram and parsing methods in terms of yield, recall, and efficiency.We then demonstrate how to use NPFST for exploratory analyses; it performs well, without configuration, on many different kinds of English text.Finally, we present a case study using NPFST to analyze a new corpus of U.S. congressional bills.For our open-source implementation, see http://slanglab.cs.umass.edu/phrases/.