Language Acquisition and Data Compression
Jason L. Hutchens, Michael D. Alder · 1997
Statistical data compression requires a stochastic language model which must rapidly adapt to new data as it is encountered. A grammatical inference engine is introduced which satisfies this requirement; it is able to discover structure in arbitrary data using nothing more than the predictions of a simple trigram model. We show that compression may be used as an alternative to perplexity for language model evaluation, and that the information processing techniques employed by our system may reflect what happens in the human brain. 1 Introduction Grammatical inference is the process of programming a computer to automatically infer a grammar for a language [8]. We consider a grammar to be nothing more than a model for some data. Applications such as speech recognition and data compression require a stochastic language model, and well-defined performance measures exist for such models. It is easy to get caught in the trap of building complicated models which utilise various ad hoc techni...