Search Abstracts | Symposia | Slide Sessions | Poster Sessions
From pixels to semantics: a realistic predictive coding model of evoked responses during reading
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Marijn van Vliet1, Laura Rautiainen1, Gayane Ghazaryan1, Samer Nour Eddine2, Aino Saranpää1, Gina Kuperberg3,4, Riitta Salmelin1; 1Aalto University, Espoo, Finland, 2University of Maryland, Maryland, USA, 3Tufts University, Medford, USA, 4Massachusetts General Hospital, Boston, USA
Decades of EEG and MEG studies of reading have tracked the cortical evoked responses following the presentation of a written word millisecond by millisecond. The now familiar timecourse is characterized by an initial visual response, followed by a series of responses that occur later and further along the ventral stream towards the temporal areas. It has been theorized that an evoked response reflects the build-up and decay of prediction error, as top-down predictions of the stimulus clash with the bottom-up perception of the stimulus. Predictive coding models have been shown to produce brain-like evoked responses, but only in unrealistically simplified scenarios, for example operating on a "letter bank" as input, and producing arbitrarily chosen semantic representations. In order to be able to simulate evoked responses that can be compared with actual neuroimaging data, we sought to construct a predictive coding model that performs the entire neural processing pipeline from extracting early visual features to higher level lexical and semantic features in a somewhat realistic form. The model follows a convolution architecture, with the input consisting of a bitmap image of written text and the output being a non-negative, sparse version of word2vec semantic word embeddings. The training set for the model consisted of different versions of 2264 possible Finnish words (15 possible fonts, size 11-36 pixels, either in lower or uppercase). We compared the simulated timecourse to actual MEG data in a series of experiments where the same stimuli were presented to both human volunteers in the scanner and the model using the same paradigm, exploring 8 types of priming. The comparison was made both qualitatively by judging the location and magnitude of experimental effects, and quantitatively through explained variance. The first experiment consisted of 800 word-pairs, where the task was to judge whether the word-identity was the same. The second word of each pair was chosen to create five priming conditions, targeting different stages of the recognition pipeline: 1) an exact copy of the first word, 2) a copy of the first word with its horizontal position shifted to the right, 3) an uppercase version of the first word, 4) a word with a large orthographic overlap with the first word, or 5) an unrelated word. For 20 participants, the inter-stimulus interval (ISI) was 700ms, and for another 20 participants, the ISI was 300ms. The second experiment consisted of 840 word-triplets (25 participants, ISI 400ms, judgment of semantic relatedness task), where the first two words were closely semantically related, and the relationship between the second and third words varied: 6) not related, 7) moderately related, or 8) closely related. Within the layers of the model, there is a rapid build-up and ramping down of prediction error that closely resembles the cortical timecourses of the grand-average MEG evoked responses. Through this direct comparison between simulated and real data, we demonstrate the merits of predictive coding as a unifying theory linking evoked responses to a mechanistic account of the cognitive processing of written stimuli.
Topic Areas: Computational Approaches, Reading