Unsupervised Multiword Segmentation of Large Corpora using Prediction-Driven Decomposition of n-grams
Julian Brooke, Vivian Tsang, Graeme Hirst, Fraser Shein · 2014
We present a new, efficient unsupervised approach to the segmentation of corpora into multiword units. Our method involves initial decomposition of common n-grams into segments which max-imize within-segment predictability of words, and then further refinement of these segments into a multiword lexicon. Evaluating in four large, distinct corpora, we show that this method cre-ates segments which correspond well to known multiword expressions; our model is particularly strong with regards to longer (3+ word) multiword units, which are often ignored or minimized in relevant work. 1