Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Language background modulates left middle frontal gyrus alignment with a multimodal language model during bilingual visual object and word processing
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Bo CHAI1, Lily Tian JIANG1, Xingzhen WANG1, Ruwen ZHANG1, Nizhuan WANG1, Wai Ting Siok1; 1The Hong Kong Polytechnic University
Introduction A recent advance in cognitive neuroscience is the use of large language models as computational architectures to model and understand human brain function. This approach enables researchers to quantify how such artificial systems mirror biological neural processing. Our study applies this method to investigate how language background shapes cortical representational geometry during bilingual visual processing. Specifically, we tested whether the representational alignment between a multimodal language model and brain activation in the left middle frontal gyrus (LMFG) – a known hub for executive control and Chinese reading – varies across participant groups, task languages, and model representational stages. Methods 41 native Chinese adults (L2 English) and 27 native English adults (L2 Chinese) underwent fMRI during separate Chinese and English task blocks. In each language, they completed six visual judgment conditions: orthographic, phonological, semantic, size, line pattern, and figure pattern. Each trial presented two stimuli with a task prompt (visually/phonologically/ semantically similar characters/words, same/different font size, line pattern, or figure pattern), followed by a binary yes/no response. First-level GLM analyses yielded condition-wise beta estimates. For each participant, response patterns were extracted from an anatomically defined, atlas-based LMFG and used to construct representational dissimilarity matrices (RDMs) separately for Chinese-task and English-task data. The same task stimuli and prompts were presented to Qwen2-VL-2B. From the vision encoder, we extracted token representations for the two input images and derived four feature spaces from them: their sum, their difference, and the token representations of the first and second images separately. After concatenating image tokens with the text prompt and passing them through the language model, we extracted image-token, text-token, and combined image-text sequence representations from layers 1, 14, and 28. Model RDMs were computed across the six conditions for each feature space and task language. Neural and model RDMs were compared using Spearman representational similarity analysis (RSA). The resulting RSA values were then analyzed with linear mixed-effects models including participant group, model condition, and their interaction, with random intercepts for participants. Results LMFG alignment with the multimodal model varied across feature spaces and differed by participant group. For English-task neural RDMs, Chinese participants generally showed higher RSA than English participants across several model conditions, and the Group × Condition interaction was significant (F(12,858)=4.281, p=1.30e-6). For Chinese-task neural RDMs, this pattern broadly reversed, with English participants showing higher RSA in several conditions, again with a significant Group × Condition interaction (F(12,858)=2.043, p=0.018). These effects were most evident in selected feature spaces, especially text-related representations, rather than uniformly across all model components. Conclusion These findings indicate that LMFG–model alignment during bilingual visual judgments is jointly modulated by language background and task language. Rather than showing a fixed correspondence to a single model layer or modality, LMFG appears to align preferentially with representational spaces that encode task structure and stimulus-comparison information. The crossed group-by-task pattern suggests that LMFG–model alignment is stronger when judgments are made in the relatively less familiar or more effortful language, consistent with accounts of LMFG as supporting explicit task structuring and control-related processing.
Topic Areas: Multilingualism, Computational Approaches