Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Shared spatial-temporal dynamics of narrative comprehension in reading and listening
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Jiawei Li1, Yuhan Lu2,3,4, Junxi Chen5, Adrien Doerig1,6, Lucia Melloni7, Xing Tian2,3,4, Radoslaw Martin Cichy1; 1Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany, 2Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai, China, 3NYU-ECNU Institute of Brain and Cognitive Science, New York University Shanghai, Shanghai, China, 4Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, Division of Arts and Sciences, New York University Shanghai, Shanghai, China, 5Department of Neurosurgery, Guangdong Sanjiu Brain Hospital, Guangzhou, China, 6Bernstein Center for Computational Neuroscience, Berlin, Germany, 7Predictive Brain Department, University Alliance Ruhr, Faculty of Psychology, Ruhr-Universität Bochum, Germany
During both reading and listening, narrative comprehension operates across multiple temporal and representational scales, integrating early sensory inputs into higher-level semantic representations. However, to which degree reading and listening share similar neural dynamics remains unclear. To investigate, we recorded stereoelectroencephalography (sEEG) data from 29 neurosurgical patients (11 female, 18 male, mean age 21.17 ± 6.35 years) as they listened to or read two short narratives (570 and 526 words). During listening, participants fixated on a central red cross while the narrative was played. During reading, participants were presented with single words on-screen, with display durations matching their audio lengths. We identified 141 active contacts during listening, which were highly centralized around the bilateral temporal region. We identified 87 active contacts during reading, which were more broadly distributed, covering temporo-occipital areas and the inferior frontal gyrus. Notably, 16 contacts showed co-activation across both modalities, located primarily in the superior temporal gyrus (STG) and central regions. We used an encoding model approach with embeddings from visual (convolutional neural network), auditory (mel spectrogram) and semantic (GPT-2) computational models to reveal the time course with which respectively visual, auditory, and semantic representations emerge with respect to word onset. We first applied the encoding approach to listening and reading data and their respectively activated 16 contacts separately. We found a strikingly similar results pattern. All three models uniquely predicted neural responses (i.e. when partialling out the effect of the other two models), but with different time courses: semantic representations peaked at ~600 ms, auditory representations peaked ~0 ms, and visual representations at ~500 ms. This suggests that similar semantic representations emerge regardless of the sensory modality with which the stimulus is delivered. It further shows that sensory (i.e. auditory or visual) features are also evoked even when the stimulus is delivered in the other modality, suggesting cross-modal generative processes in reading and listening. To test whether similar representations emerge in both reading and listening directly, we applied the encoding approach across sensory modalities, i.e. training the encoding model on one modality, and testing it on the other, on the 16 co-activated contacts. We observed a result pattern equivalent to the one reported above. Together, our findings demonstrate that auditory, visual, and semantic representations are robustly shared within the STG and central regions during both reading and listening.
Topic Areas: Computational Approaches,