Word Class and Syntax Rule Representations Spontaneously Emerge in Recurrent Language Models
Patrick Krauß, Kishore Surendra, Paul Stoewer, Andreas Maier, Claus Metzner, Achim Schilling · 2025
How do humans learn language, and can the first language be learned at all? These fundamental questions are still hotly debated. In contemporary linguistics, there are two major schools of thought that give completely opposite answers. According to Chomsky&s;s theory of universal grammar, language cannot be learned because children are not exposed to sufficient data in their linguistic environment. In contrast, usage-based models of language assume a profound relationship between language structure and language use. In particular, contextual mental processing and mental representations are assumed to have the cognitive capacity to capture the complexity of actual language use at all levels. The prime example is syntax (i.e., the rules by which words are assembled into larger units such as sentences). Typically, syntactic rules are expressed as sequences of word classes. However, it remains unclear whether word classes are innate, as implied by universal grammar, or whether they emerge during language acquisition, as suggested by usage-based approaches. Here, we address this issue from a machine learning and natural language processing perspective. In particular, we trained two deep recurrent neural networks (RNNs) consisting of bi-directional long-short-term-memory layers on predicting the next word, provided sequences of consecutive words as input in two different languages, and a further RNN on predicting the next two words. Subsequently, we analyzed the emerging activation patterns in the hidden layers of the RNNs. Strikingly, we find that the internal representations of nine-word input sequences in both languages cluster according to the word class of the tenth word to be predicted as output, even though the neural network did not receive any explicit information about syntactic rules or word classes during training. Furthermore, in the RNN trained on predicting the next two words, we find that the representations of input sequences cluster according to the word class combinations of the two words to be predicted as output. These surprising results suggests, that also in the human brain, abstract representational categories such as word classes or combinations of word classes (i.e., syntax rules) may naturally emerge as a consequence of predictive coding and processing during language acquisition.