Investigating semantic expectation and predictive error in the visual cortex with a large fMRI vision-language dataset
Shurui Li, Zheyu Jin, Ru‐Yuan Zhang, Shi Gu, Yuanning Li · Journal of Vision · 2025
Classical models of visual processing in the brain emphasize a predominantly feedforward hierarchical coding scheme, where lower-level features are progressively integrated into higher-level semantic representations. However, this view fails to fully account for the complex and dynamic nature of semantic information processing in the visual cortex, which involves interactions that extend beyond passive feedforward pathways. To study the neural coding of semantic information in the visual hierarchy, we collected a large-scale fMRI vision-language dataset, where each participant processed over 4,400 paired stimuli consisting of a text caption followed by a naturalistic image, with the task of evaluating their semantic congruence. Driven by the predictive coding theory, we hypothesized that the early visual cortex can represent semantic expectations and predictive errors. First, we observed that when subjects were presented with images that matched their semantic expectations, the early visual cortex responded significantly less than they were presented with unexpected images. To explain the neural coding, we built neural encoding models using features extracted from large language and vision models. We found that language model features could predict the early visual cortex’s response after the subjects viewed a text caption. This indicates that the early visual cortex can generate semantic expectations. Next, we found that neural activity from V1 to V3 encode prediction mismatch of visual stimuli, representing a progression from low-level to high-level prediction errors. Finally, we found that the degree of response amplitude reduction correlated to the neural coding of high-level prediction errors. To sum up, using a large fMRI vision-language dataset, we provide evidence of cross-modal semantic expectation and predictive error coding in the visual cortex.