Search Abstracts | Symposia | Slide Sessions | Poster Sessions
A joint language-vision code for grounded language learning
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Shaoyun Yu1, Gangyi Feng1; 1The Chinese University of Hong Kong
The symbol grounding problem (Harnad, 1990) raises a long-standing question about the nature of language representations. Can linguistic representations derived purely from distributional statistics capture meaning, or must they be grounded in sensorimotor experience to do so? Modern large language models (LLMs) are, at their core, still language-only learners (Tong et al., 2026), exemplifying Harnad’s metaphor of “learning Chinese from a Chinese dictionary alone.” Although language-only model embeddings have been shown to successfully predict cortical responses (Schrimpf et al., 2021; Goldstein et al., 2022; Yu et al., 2024), the evidence is largely confined to unimodal listening or reading tasks in fluent native speakers. To move from the endpoint of language processing to the origin of meaning, we ask whether the cortical signatures of language learning are explained by language-only representational geometry, or whether acquiring meaning requires the brain to form a joint language-vision code that integrates linguistic information with perceptual experience. To simulate multimodal language learning in adults, we conducted a seven-day artificial language learning experiment in which thirty-two participants learned Brocanto2, a constructed language whose grammar is modeled after natural languages and whose use is grounded in a visual board game (Morgan-Short et al., 2012; Feng et al., 2021). Participants underwent fMRI scanning on days 1, 2, 3, and 7 while being exposed to game images paired with corresponding phrase- and sentence-level language stimuli. We compared the representational geometry of learners’ brain responses with three types of model representations: (1) language-only representations extracted from a custom GPT-2 model trained on the Brocanto2 corpus, (2) visual representations for the game states derived from Meta DINO v3 (Siméoni et al. 2025; Adeli et al., 2026), and (3) a combination of the language and vision representations. We found that joint language-vision representations best captured learners’ neural responses to the newly learned language, surpassing unimodal model-brain alignment across visual, sensorimotor, attention, cognitive control, and auditory-language processing networks. Variance partitioning indicated complementary contributions from language and vision: each modality uniquely accounted for at least 25% of explained variance in 90% of ROI-by-day cases. Permutation tests further revealed that unique contributions of language-only representations were most robust on day 2 in visual, attention, control, and default-mode networks, whereas the unique explanatory power of visual representations was concentrated in the visual cortex, with its effect persisting from day 1 to day 7. These findings suggest that the neural representation of a newly learned language cannot be fully explained by a single modality. Linguistic representations captured the distributed higher-order structure that emerged as learners began to acquire the language’s regularities, whereas visual representations provided a persistent grounding scaffold for mapping utterances onto meaningful perceptual scenes. Language learning is therefore best characterized as the formation of grounded multimodal representations that link linguistic structure with perceptual experience.
Topic Areas: Language Development/Acquisition, Computational Approaches