Poster Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions

Surprisal and semantic distance index shared and distinct neural computations in naturalistic movie viewing

Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai

Ryan M. O Leary^1,2, Hailey C. Smith^1,3, Emily B. Myers^4,5,6, Jamie Reilly^7,8, Jonathan E. Peelle^1,2,3; ¹Institute for Cognitive and Brain Health, Northeastern University, ²Department of Communication Sciences and Disorders, Northeastern University, ³Department of Psychology, Northeastern University, ⁴Department of Psychology, University of Connecticut, ⁵Department of Speech, Language, and Hearing Sciences, University of Connecticut, ⁶Institute for Brain and Cognitive Sciences, University of Connecticut, ⁷Department of Communication Sciences and Disorders, Temple University, ⁸Department of Psychology, Temple University

Comprehension of natural speech occurs rapidly over time, requiring listeners to integrate incoming acoustic input with prior linguistic and thematic knowledge to access meaning. Previous studies using large language models to estimate word predictability through surprisal values have demonstrated that surprisal drives broad activation of core language regions. However, surprisal reflects both syntactic and semantic information, making it impossible to clearly isolate which component drives neural responses. In contrast, semantic distance can examine how closely a word aligns with previous words in high-dimensional semantic space, offering a more selective method to isolate the semantic component of contextual integration. Previous studies have shown that both surprisal and semantic distance are associated with activity in bilateral superior temporal gyrus and the left inferior frontal gyrus. While these measures have previously been studied in isolation, the present study directly contrasted surprisal and semantic distance during naturalistic audiovisual language comprehension. We analyzed fMRI scans from 20 English speaking adults from the Naturalistic Neuroimaging Database who viewed the full-length feature film 500 Days of Summer (~95 minutes) during continuous scanning. For each word in the movie, semantic distance and surprisal was calculated from the film transcript using a context window of previous words. We implemented a series of voxelwise encoding models which were hierarchically organized to test the incremental addition of feature sets within the movie to see what remaining variance semantic distance and surprisal explain after accounting for lower-level features. The first model included a suite of visual features. The second model added the auditory amplitude envelope, and the third model added word-level features such as lexical frequency and phonological neighborhood density. The final model included features from all the previous models as well as semantic distance and surprisal. Incremental models were used to partition unique contributions. Group-level significance was determined using nonparametric permutation testing with threshold-free cluster enhancement. The model containing visual features primarily predicted activity in occipital regions. The addition of the auditory amplitude envelope extended model predictivity into bilateral superior temporal gyrus. Word level features further increased predictivity in bilateral superior temporal gyrus, and increased predictivity in frontal regions. The addition of surprisal and semantic distance further improved prediction in bilateral superior temporal gyrus and left inferior frontal gyrus beyond what was predicted from previous models. However, model comparisons that accounted for shared variance between surprisal and semantic distance revealed a partial dissociation where surprisal improved prediction in inferior frontal gyrus and superior temporal gyrus, while the unique contribution of semantic distance was predominantly constrained to bilateral superior temporal gyrus. These results support the notion that the brain tracks broad probabilistic prediction as well as semantic integration during naturalistic language comprehension. Specifically, the superior temporal gyrus appears to be sensitive to both overall predictability and semantic similarity, while the left inferior frontal gyrus may reflect broader demands associated with integrating unexpected words. More generally, these findings extend previous work on auditory-only speech by demonstrating that surprisal and semantic distance can predict meaningful neural variation even within in a rich audiovisual context.

Topic Areas: Speech Perception, Computational Approaches