Coordinate structure constraint in the linguistic competence of large language models
Pavel Grashchenkov, Kseniia A. Studenikina, Lada I. Pasko · Vestnik of Saint Petersburg University Language and Literature · 2024
A syntactic island is a construction extraction from which leads to ungrammaticality. Island constraints are generally demonstrated through the impossibility of the A′-movement, e. g. wh-movement. Considering extraction from an island as ungrammatical is common to all native speakers. In terms of natural language understanding and generation, the competence of large language models (LLM) is almost indistinguishable from the human one. However, the difference between the grammatical constraints of the native speakers and LLM are still studied insufficiently. If the LLM grammar is set up similar to the human one, they will demonstrate high sensitivity to island constraints. The current study aims to compare the language competence of the native speakers and LLM based on the coordinate structure islands. The three dialogue systems — ChatGPT, YandexGPT and GigaChat — were examined via two tests. The first one investigates whether the model is able to give a semantically correct answer to the question with violation of island constraints. The second test directly accesses the grammaticality judgements. The results clearly show that the LLM language competence differs from the human one. The observed models regularly answer the questions violating island constraints correctly and consider them grammatical. YandexGPT turns out to be more consistent, while ChatGPT and GigaChat frequently give incorrect answers to the questions Яwhich they judge acceptable. The influence of the stimuli’s grammatical features depends on the model: the island sensitivity of ChatGPT and GigaChat is determined by the same features in contrast to YandexGPT. Thus, the results call into question the fact that LLM language competence is close to the human one.