Large Language Models are Not Models of Natural Language: They are Corpus Models
Csaba Veres · IEEE Access · 2022
Natural Language Processing (NLP) has become one of the leading application areas in the current Artificial Intelligence boom.Transfer learninghas enabled largedeep learningneural networks trained on thelanguage modelingtask to vastly improve performance in almost all language tasks. Interestingly, when the models are trained with data that includes software code, they demonstrate remarkable abilities in generating functioning computer code from natural language specifications. We argue that this creates a conundrum for claims that neural models provide an alternative theory togenerative phrase structure grammarsin explaining how language works. Since the acceptable syntax of programming languages is determined by phrase structure grammars, successful neural models are apparently uninformative about the theoretical foundations of programming languages, and by extension, natural languages. We argue that the termlanguage modelis misleading because deep learning models are not theoretical models of language and propose the adoption ofcorpus modelinstead, which better reflects the genesis and contents of the model.