Fixation-related brain activity during semantic integration of object-scene information
Moreno I. Coco, Antje Nuthmann, Olaf Dimigen · 2018
In vision science, a topic of great interest and considerable controversy is the processing of objects that are (in)consistent with the overall meaning of the scene in which they occur. How quickly can we access the semantic properties of objects and does this happen before the object is directly looked at? Here we brought novel evidences to this debate by co-registering eye-movements and EEG while participants freely explored photographs of indoor scenes. Each scene contained a target object that was either consistent or inconsistent with the scene context (e.g., toothpaste vs. flashlight in a bathroom). Eye-movement behaviour showed that inconsistent objects were not only more effortful to process (i.e., looked at longer), but also prioritized over consistent objects while they were still in extrafoveal vision (i.e., looked at earlier after scene onset). In fixation-related brain potentials (FRPs), we used a linear deconvolution technique to compare the activity elicited by semantically consistent versus inconsistent objects at scene onset (i.e. in the stimulus-locked ERP), during the target fixation (t), and during the preceding fixation (t-1). We demonstrate a fixation-related N300/N400 effect, whereby inconsistent objects elicited a larger negative shift of FRP activity than consistent objects, already at the pre-target fixation t-1 and throughout fixation t. Taken together, our results suggest that object meaning can be, at least partly, extracted in extrafoveal vision, and highlight how attentional and neural mechanisms can be simultaneously investigated to uncover the mechanisms of object-scene integration during natural viewing.