A Random Text Model for the Generation of Statistical Language Invariants
Chris Biemann · Publikation Server (Leipzig University) · 2007
Abstract A novel random text generation model is introduced. Unlike in previous random text models, that mainly aim at producing a Zipfian distribution of word frequencies, our model also takes the properties of neighboring co-occurrence into account and introduces the notion of sentences in random text. After pointing out the defi-ciencies of related models, we provide a generation process that takes neither the Zipfian distribution on word frequencies nor the small-world structure of the neighboring co-occurrence graph as a constraint. Nevertheless, these distribu-tions emerge in the process. The distribu-tions obtained with the random generation model are compared to a sample of natu-ral language data, showing high agree-ment also on word length and sentence length. This work proposes a plausible model for the emergence of large-scale characteristics of language without as-suming a grammar or semantics. 1 Introduction G. K. Zipf (1949) discovered that if all words in a sample of natural language are arranged in de-creasing order of frequency, then the relation be-tween a words frequency and its rank in the list follows a power-law. Since then, a significant amount of research in the area of quantitative lin-guistics has been devoted to the question how this property emerges and what kind of processes gen-erate such Zipfian distributions. The relation between the frequency of a word at rank r and its rank is given by f(r) µ r