The spatio-temporal representation of natural reading
Leila Wehbe · International Conference on Artificial Intelligence · 2015
My thesis is about studying how the brain organizes complex information when it read text in a naturalistic setting. My work is an integrated interdisciplinary effort which employs functional neuroimaging, and revolves around the development of machine learning methods to uncover multilayer cognitive processes from brain activity recordings. Studying how the human brain represents meaning is not only important for expanding our scientific knowledge of the brain and of intelligence. By mapping behavioral traits to differences in brain representations, we increase our understanding of neurological disorders that plague large populations, which may bring us closer to finding treatments (as detailed in the last section of this statement). For all these purposes, functional neuroimaging is an invaluable tool. Traditional functional neuroimaging studies typically consist of highly controlled experiments which vary along a few conditions. The stimuli for these conditions are artificially designed, and therefore might result in conclusions that are not generalizable to how the brain works in real life. When studying language processing for example, very few experiments show subjects a real text, and show instead carefully designed stimuli. Furthermore, the analysis of functional neuroimaging data has typically consisted in simple comparisons: regions which respond differently to the individual conditions are identified. Many researchers have recently started using brain decoding (i.e. classifying the stimulus being processed from the subject’s brain image), which can reveal responses encoded in subtle patterns of activity across a brain region. However, brain decoding is still mostly used in a rather limited fashion. In order to predict which condition an image corresponds to, a classifier is trained on several examples of each condition. This classifier is not able to generalize its knowledge to novel conditions not seen in training. It can therefore be argued that such a model does not represent a broad understanding of brain function. This work revolves around studying the parallel cognitive processes involved when subjects are engaged in a naturalistic language processing task, namely reading a chapter of a real book. We use computational linguistics algorithms to model the content of the text, and machine learning to identify regions in the brain that are involved in processing its different components. This work is consequently an integrated interdisciplinary effort. The spatial representation of language subprocesses We set out to challenge the understanding that it is difficult to study the complex processing of natural stories. We used functional Magnetic Resonance Imaging (fMRI) to record the brain activity of subjects while they read an unmodified chapter of a popular book. Unprecedently, we modeled the measured brain activity as a function of the content of the text being read Wehbe et al. [2014a]. Our model is able to extrapolate to predict brain activity for novel passages of text beyond those on which it has been trained. Not only can our model be used for decoding what passage of text was being read from brain activity, but it can also report which type of information about the text (syntax, semantic properties, narrative events etc.) is encoded by the activity of every brain region. Using this model, we found that the different regions that are usually associated with language are processing different types of linguistic information. We are able to build detailed reading representations maps, in which each region is labeled by the type of information it processes in the distributed task of story understanding.