Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Contextualized Representations from LLMs Explain Unique Variance in Language Processing: Evidence from EEG and fMRI
Poster Session F, Friday, October 2, 2:45 - 4:45 pm, Wangari Maathai
Bei Xiao1, Xufeng Duan1, Yibo Wang1, Zhenguang G. Cai1; 1The Chinese University of Hong Kong
Quantitative predictors of human language processing can be extracted from Large Language Models (LLMs), but it remains unclear whether these predictors explain neural responses beyond classical measures such as cloze probability. Moreover, language comprehension unfolds within continuous discourse, yet LLMs are often applied only to isolated stimulus sentences when predicting neural responses. This leaves open whether matching the LLM input to the discourse history actually experienced by participants improves prediction. We address two questions in the study: (1) Do LLM representations explain unique variance in the N400 beyond cloze probability? (2) Do discourse-contextualized LLM representations explain unique variance beyond their sentence-only counterparts? Methods. We analyzed eight English N400 datasets (Lago et al., 2019; Ryskin et al., 2021; Nieuwland et al., 2018; Szewczyk & Federmeier, 2022), comprising 492 participants and 58,353 noun-locked epochs. Cloze norms were available for six datasets, covering 77.5% of epochs. Surprisal was extracted from Llama-3.2-3B and Qwen2.5-7B. Linear mixed-effects models predicted single-trial N400 from surprisal with subject and item as random effects. We used likelihood-ratio tests to compare nested models and evaluate the unique contribution of each predictor. Representation similarity analysis (RSA, correlation between neural and surprisal-based representational dissimilarity matrices) is performed to investigate the questions from the representation geometry perspective. For fMRI data, hidden states of Pythia-410m from 31 checkpoints were mapped via ridge regression to language-related regions on Blank (2014) and Pereira (2018). LLM surprisal was defined as −log2P(w | preceding tokens), where w is the target word; the two surprisal variants differed in what counted as “preceding tokens”. Sentence-only surprisal was computed from the target sentence alone, whereas contextualized surprisal included all prior sentences seen by participants. For fMRI, the same contrast was applied to hidden states extracted from the target sentence alone versus the full story. Results. LLM surprisal explained unique N400 variance beyond cloze in both LMMs (χ²(1) = 374/381 for Llama/Qwen, p < .001) and RSA after partialling out cloze (ρ = .32/.31, p < .001). Moreover, contextualized surprisal explained additional N400 variance beyond cloze probability (χ²(1) = 396/401, p < .001) and likewise remained a strong predictor in RSA after controlling cloze (ρ = .33/.34, p < .001). Contextualized surprisal carried information that partially overlapped but was not identical to sentence-only surprisal: the two each contributed unique variance (contextualized: χ²(1) = 21/32, p < .001). Dataset-level comparisons slightly favoured contextualization (e.g., in the Wlotko & Federmeier (2022) dataset, contextualized surprisal significantly improved prediction beyond cloze with Llama, whereas sentence-only surprisal did not). Similarly, in fMRI, contextualized hidden states aligned more strongly with language-network activation than sentence-only (β = .011, p = .042). LLM surprisal robustly predicts N400 amplitude beyond cloze probability. Specifically, contextualized surprisal captures variance beyond cloze. Contextualized LLM representations carry information not captured by sentence-only representations, at least in current datasets. These findings suggest that contextualized LLM representations may provide a useful model of how the brain integrates language beyond the current sentence.
Topic Areas: Computational Approaches, Meaning: Discourse and Pragmatics