Structure Learning in Weighted Languages
András Kornai, Attila Zséder, Gábor Recski · SZTAKI Publication Repository (Hungarian Academy of Sciences) · 2013
We present Minimum Description Length techniques for learning the structure of weighted languages.MDL is already widely used both for segmentation and classification tasks, and here we show it can be used to formalize further important tools in the descriptive linguists' toolbox, including the distinction between accidental and systematic gaps in the data, the detection of ambiguity, the selective discarding of data, and the merging of categories.