Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Two Cortical Mechanisms for Natural Audiovisual Processing
Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai
Subha Nawer Pushpita1, Leila Wehbe1; 1Carnegie Mellon University
Understanding how human brains process naturalistic audiovisual information remains a central challenge in cognitive neuroscience. Progress has been limited by difficulties of modeling complex audiovisual features - most prior work has relied on short, controlled stimuli or a single modality, leaving real-world comprehension mechanisms poorly characterized. Although recent advances in AI have enabled the extraction of high-quality features, how cortical regions concurrently process auditory and visual information as the information load between the modalities varies across time remains largely unexplored. Using large-scale fMRI data collected while participants watched Hollywood movies, released as part of the Algonauts 2025 Challenge, we developed two novel computational approaches that leverage the timepoint-wise prediction performance of modality-specific encoding models to map sensory dynamics across the cortex. The first approach detects sustained periods of audio- or video-dominance in prediction performance — i.e., when one modality's encoding model substantially outperforms the other — identifying regions that dynamically switch the modality they encode over time; the second isolates extended periods when both modalities predict a region well, revealing regions that maintain simultaneous encoding of both. The auditory encoding model was fit on Qwen2-Audio representations of the movie's audio track, and the visual encoding model on Qwen2.5-VL and VideoMAE representations of the video track. The first approach revealed a striking organization of “switching” regions forming two distinct bands: a posterior band encircling visual cortex - including the precuneus, superior parietal lobule, angular gyrus, anterior lateral occipital cortex, and temporo-occipital divisions of the middle and inferior temporal gyri, and an anterior band spanning lateral frontal association areas, including parts of the superior, middle, and inferior frontal gyri. The second approach revealed a continuous axis of regions, simultaneously predicted by both modalities, that extends from the lateral occipital cortex into the temporal cortex. We validate these findings in two ways. First, human raters confirm that scenes identified as audio or video-dominated by our method are ones in which the corresponding modality is more relevant to understand the content. Second, we develop an audio-visual transformer that predicts fMRI responses, and its learned time-dependent attention-distribution over audio and visual inputs across regions matches the two patterns identified through our approaches. We further show that this dominance is not driven by a lack of information in the non-dominant modality — its features remain as rich as usual — but rather by the brain substantially reducing how much it encodes them, suggesting active sensory reweighting at those moments. We also show that modality dominance in switching regions cannot be simply explained by the dominant modality carrying the region's preferred concepts that can in principle be conveyed by either modality; regions do not merely track whichever modality happens to express them. The coexistence of these systems points to a cortical architecture that adaptively reweights sensory inputs in some regions while maintaining balanced multimodal representations in others, supporting robust comprehension of complex naturalistic events. More broadly, this work demonstrates how naturalistic neuroimaging paired with modern machine learning can reveal new principles of dynamic audiovisual processing.
Topic Areas: Multisensory or Sensorimotor Integration, Computational Approaches