Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Dynamic cortical code for time-continuous tracking of speaker emotional valence
Poster Session F, Friday, October 2, 2:45 - 4:45 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Jeffrey Xing1, Benjamin Lang1, Lauren Ostrowski1, Kate Urrutia1, Travis Ramirez1, Mingxiong Huang1,2, Timothy Gentner1; 1University of California, San Diego, 2VA San Diego Healthcare System
In speech communication, paralinguistic features are voice cues that accompany language production to convey socially-relevant information (e.g., speaker emotion, identity) and augment the speaker’s intended linguistic message (Guyer et al., 2021). It remains unclear, however, how the brain abstracts paralinguistic information in parallel with linguistic information from the same underlying acoustic source. Examining speaker emotion creates a unique entry point for testing hypotheses about how paralinguistic and linguistic information are dually processed. Speaker emotion is affected by acoustic-prosodic cues that also contribute to linguistic structures (Ní Chasaide and Gobl, 2004; Scherer, 2018), and also by direct lexico-semantic structures (Ben-David et al., 2016). We will examine how the brain integrates acoustic-prosodic and lexico-semantic cues to abstract speaker emotional valence. Recent evidence suggests that linguistic information in speech is processed by a dynamic neural code, wherein linguistic information flows through neural ensembles over time and is sustained beyond the physical presentation of the speech stimulus (Gwilliams et al., 2025). We hypothesize that speaker emotional valence in speech is similarly processed by a dynamic code that flexibly integrates with the linguistic dynamic code. We investigate this hypothesis with a natural listening task using magnetoencephalography (MEG). Stimuli consist of 56 English speech excerpts (17-77s) from the Buckeye corpus (Pitt et al. 2007), wherein 10 excerpts are lowpass filtered (removes semantics) and 10 excerpts are presented as time-locked transcripts (removes acoustics). Participants (N = 65) were instructed to use a button-box to continuously move an on-screen slider throughout stimulus presentation to rate the speaker’s emotional valence. Previous behavioral results from a supporting study (N = 173) suggest that listeners flexibly weigh acoustic and lexico-semantic information at different timepoints and timescales in speech excerpts to infer speaker emotion in this task. We predict that the perceptual neural dynamics of participants who completed this task in the MEG will be similar, reflecting how the brain integrates acoustic and semantic information to form paralinguistic perception. We will first model the neural temporal dynamics of perceiving speaker emotion. To isolate perceptual dynamics from motor execution, we will first model motor-related cortical responses by fitting a temporal receptive field to the MEG data using button-press events in an independent visuomotor task. This estimated motor response will then be regressed out from the continuous rating MEG data. We will then train logistic regression decoders before rating-change events to capture the neural activity preceding emotional valence rating change. We will then quantify how these dynamics change and sustain over time using the temporal generalization method (King and Dehaene, 2014). To investigate interactions between paralinguistic and linguistic dynamic neural codes, we will perform cross-condition decoding. We will train temporal decoders on rating-change events during lowpass-filtered (acoustic-prosodic only) and time-locked transcript (lexico-semantic only) stimulus conditions and test their performance on unaltered speech. We predict that acoustic-prosodic and lexico-semantic information will exhibit different timescales of temporal generalization, and that models dynamically fluctuate in performance when tested on different timepoints of unaltered speech, potentially revealing how the brain flexibly integrates paralinguistic vs linguistic features.
Topic Areas: Speech Perception, Computational Approaches