Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Shared Neural Responses to Audiovisual Speech Across Distinct Movie Stimuli
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Hailey Smith1,2, Ryan O’Leary1,3, Jonathan Peelle1,2,3; 1Institute for Cognitive and Brain Health, Northeastern University, 2Department of Psychology, 3Department of Communication Sciences and Disorders
Speech perception is inherently multisensory, requiring us to process pitch, tone, and visual input simultaneously. Studying speech perception reveals how our senses collaborate in noisy environments, making it ideal for examining multisensory integration. Neuroimaging studies often implicate the left posterior superior temporal sulcus (pSTS) as playing a key role in multisensory integration. Traditional experiments often isolate auditory, visual, and language elements, creating a “laboratory-style” bias, and it remains unclear how the pSTS functions during continuous, natural speech. To address this, researchers increasingly use naturalistic stimuli – continuous, dynamic, and multisensory, including movie-watching paradigms. Movie-watching combines visual, auditory, and narrative elements, engaging broad neural networks and eliciting consistent sensory responses across individuals (Hasson et al., 2004). In the current study, we assessed multisensory speech perception using fMRI data from two full-length films (500 Days of Summer and Citizen Four) from the Naturalistic Neuroimaging Database (NNDb). We manually classified 22,636 words as audiovisual (12,556) or auditory-only (10,080). Classification was based on visible mouth movements and facial visibility after automated transcription alignment. We convolved these speech event onset times with a hemodynamic response function (HRF) and modeled them in SPM. Nuisance regressors included motion, faces, and visual control parameters. Group-level analyses used threshold-free cluster enhancement (TFCE), p < .05 FWE-corrected. We predicted that canonical auditory and audiovisual speech regions (e.g., STG, pSTS, visual cortex) would respond differently to auditory-only and audiovisual speech. We also hypothesized that this functional organization would remain robust under naturalistic viewing conditions, demonstrating generalizable multisensory integration beyond simplified experimental paradigms. Across 44 participants (M age = 27.7 years; 50% female), audiovisual speech processing elicited significant activation in bilateral superior temporal gyri (STG) and the posterior superior temporal sulcus (pSTS), consistent with established models of multisensory integration. However, beyond these canonical regions, audiovisual speech also recruited temporal and inferior frontal areas associated with semantic retrieval and higher-order language processing, suggesting broader network engagement during naturalistic viewing compared to laboratory paradigms. In contrast, auditory-only speech elicited greater activation in occipital and attentional networks, indicating differential allocation of visual resources in the absence of informative facial cues. Together, these findings suggest that naturalistic speech perception engages distributed multisensory and semantic systems beyond traditional laboratory-defined speech networks, emphasizing the importance of ecological validity for understanding real-world communication.
Topic Areas: Multisensory or Sensorimotor Integration, Speech Perception