Memory for Prediction: A Transformer-based Integrative Account of Sentence Processing
Ryu, Soo Hyun · Deep Blue (University of Michigan) · 2025
This thesis proposes that Transformer-based language models serve as integrative sentence processing models that combine expectation-based (e.g., surprisal theory) and memory-based (e.g., cue-based retrieval theory) accounts of sentence processing. To show that attention can estimate memory retrieval interference, I introduce a novel Transformer-based metric — attention entropy. By analyzing naturalistic sentence reading time data, I demonstrate that this new metric explains variance in sentence processing difficulty not accounted for by surprisal alone, presumably reflecting memory integration difficulty. To further validate attention entropy as an estimate of memory integration difficulty, I show that its effects on predicting sentence reading times vary depending on the speed–accuracy trade-offs imposed on participants, with the strongest effects observed in accuracy-emphasized conditions. I also show that attention entropy accounts for memory interference effects in psycholinguistic phenomena such as subject–verb agreement, embedded sentence structures, and relative clause processing. This thesis concludes with a discussion of limitations and future work.