On Hapax Legomena and Morphological Productivity
Janet B. Pierrehumbert, Ramon Granell · 2018
Quantifying and predicting morphological productivity is a long-standing challenge in corpus linguistics and psycholinguistics.The same challenge reappears in natural language processing in the context of handling words that were not seen in the training set (out-ofvocabulary, or OOV, words).Prior research showed that a good indicator of the productivity of a morpheme is the number of words involving it that occur exactly once (the hapax legomena).A technical connection was adduced between this result and Good-Turing smoothing, which assigns probability mass to unseen events on the basis of the simplifying assumption that word frequencies are stationary.In a large-scale study of 133 affixes in Wikipedia, we develop evidence that success in fact depends on tapping the frequency range in which the assumptions of Good-Turing are violated.