Search Abstracts | Symposia | Slide Sessions | Poster Sessions
From vision, sound, and language to shared representational geometry in multimodal LLMs and the brain
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Chengcheng Wang1, Zhengwu Ma1, Yike Wang1, Jixing Li1; 1City University of Hong Kong
Introduction. Large language model (LLM) embeddings have emerged as powerful computational models of human language processing. Representations from text-based LLMs have been shown to predict neural responses during naturalistic language comprehension, suggesting that these models capture aspects of the representational structure used by the human brain. However, real-world communication is inherently multimodal: speech unfolds together with vocal acoustics and broader visual context. This raises the question of whether multimodal LLMs learn modality-general representations that converge across vision, audition, and language, or do they preserve modality-specific structures that align with dissociable neural systems? Here, we examined this question using fMRI and MEG data collected during naturalistic movie watching. The same video segments were passed through a multimodal LLM, from which we extracted text, audio, and visual embeddings using the model’s corresponding modality-specific encoder. We find that Although modality-specific embeddings selectively aligned with sensory and language cortices, representations from different modalities became increasingly convergent across deeper model layers. These findings suggest that multimodal LLMs preserve modality-specific structure at early stages while progressively integrating information into a more shared representational space, providing a nuanced view of the platonic representation hypothesis. Methods. Thirty participants took part in the fMRI experiment (17 females; mean age = 23.17 ± 2.31 years), and an independent group of thirty participants took part in the MEG experiment (16 females; mean age = 22.67 ± 1.99 years). The stimulus consisted of a continuous video clip extracted from a Chinese reality TV show, with a total duration of approximately 25 minutes. We used a multimodal large language model, Qwen-2.5-Omni-7B (Xu et al., 2025), to extract modality-specific representations from the stimulus. Specifically, embeddings were extracted from video frames, audio waveforms, and transcribed words using the model’s corresponding visual, auditory, and textual encoders. These embeddings were then aligned with neural responses using ridge regression. For the fMRI data, embeddings were time-locked to word onsets and used to predict BOLD responses at each voxel. To examine the temporal dynamics of model–brain alignment, we applied a similar encoding approach to source-localized MEG data within the significant cortical clusters identified from the fMRI analyses. For each source location and time point, embeddings were regressed against MEG activity within a −100 to 500 ms window surrounding word onset. Group-level statistical significance was assessed using cluster-based permutation tests (Maris & Oostenveld, 2007) with 10,000 permutations to identify significant spatiotemporal clusters. Results. Modality-specific embeddings showed partially dissociable but systematically organized patterns of model–brain alignment. Visual embeddings primarily predicted activity in occipital and ventral temporal regions, whereas audio embeddings showed stronger alignment with superior temporal regions. Text embeddings, in contrast, preferentially aligned with temporal regions and the angular gyrus. Additionally, in lower layers, text, audio, and visual embeddings occupied relatively distinct representational spaces, reflecting the modality-specific information carried by each input stream. However, cross-modal representational similarity increased progressively in deeper layers. This convergence suggests that the model gradually transforms modality-specific inputs into a more shared representational format, integrating linguistic, auditory, and visual information at higher levels of abstraction.
Topic Areas: Computational Approaches, Multisensory or Sensorimotor Integration