Search Abstracts | Symposia | Slide Sessions | Poster Sessions
LLM-derived emotion vectors predict MEG responses beyond acoustics and word embeddings
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Zhengwu Ma1, Jixing Li; 1City University of Hong Kong
Introduction. During naturalistic listening, listeners process multiple types of information simultaneously, including acoustic signals, word meanings, and emotional content. Previous encoding studies have shown that acoustic features and word embeddings from large language models (LLMs) can reliably predict brain responses during narrative comprehension. However, it remains unclear whether the brain also tracks higher-level emotional information that cannot be fully explained by acoustics or lexical-semantic embeddings. Recent work in mechanistic interpretability suggests that emotion-related representations can be extracted from LLM activations (Sofroniew et al., 2026). In this study, we adapted an emotion-probe approach to quantify word-level emotional information in a Mandarin Chinese audiobook version of The Little Prince. By comparing emotion-based encoding with acoustic and word-embedding models, we investigated whether emotional representations explain residual MEG responses during naturalistic story listening. Methods. We recorded MEG data from 32 right-handed native Mandarin speakers (16 females, mean age=26.6 years, SD=3.9), who passively listened to a Mandarin Chinese audiobook version of The Little Prince. We first computed source-localized inter-subject correlation (ISC) to identify cortical responses reliably driven by the narrative. To isolate higher-level residual signals, we regressed out acoustic features, including pitch, intensity, envelope, and spectrogram, followed by word-level embeddings extracted from Qwen3-8B (Yang et al., 2025), and recomputed ISC on the residual responses. To quantify emotional information in the stimulus, we adapted the emotion-probe approach of Sofroniew et al. (2026). We used the 100 human-written topics from Sofroniew et al. as narrative premises. For each topic, we used Qwen3-8B to generate 12 emotion-targeted Chinese stories, corresponding to 12 core emotions: afraid, angry, calm, desperate, guilty, happy, inspired, loving, nervous, proud, sad, and surprised. Residual-stream activations were extracted for each story at every model layer and averaged across tokens. For each layer, this yielded 12 emotion-specific vectors corresponding to the 12 targeted emotion categories. Each word in The Little Prince was represented by its cosine similarity to the resulting 12 emotion vectors. For each participant and cortical vertex within the ISC mask, we trained a linear regression model on the first eight story sections and tested them on the final section. Prediction accuracy was measured using Pearson correlation, and group-level clusters were assessed using 10,000-permutation cluster-based tests (Maris & Oostenveld, 2007). Results. We first validated the emotion probe by assessing emotion separability across model layers. Layer 15 produced the largest margin between the highest and second-highest emotion similarities across words; therefore, we used the layer-15 emotion vectors for the neural encoding analysis. Raw ISC revealed broad stimulus-driven responses during naturalistic listening. After acoustic features and word embeddings were regressed out, residual ISC was reduced but remained evident in the left superior temporal and inferior frontal regions, as well as the right superior temporal gyrus. Emotion vectors significantly predicted MEG responses in the bilateral superior temporal gyri. Notably, right-hemisphere emotion encoding closely resembled the residual ISC pattern, whereas left-hemisphere emotion encoding showed greater overlap with acoustic and word-embedding maps. These results suggest that emotional information is continuously tracked during narrative comprehension beyond acoustic and lexical-semantic representations.
Topic Areas: Computational Approaches, Multisensory or Sensorimotor Integration