Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Emotional prosody reveals a complex relationship between cortical envelope tracking and speech recognition in full-spectrum and vocoded speech
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Jessica Alexander1, Fernando Llanos1; 1The University of Texas at Austin
Cortical tracking of the speech envelope has been widely proposed as a neural mechanism supporting speech intelligibility. In studies that manipulate intelligibility through acoustic degradation, improved recognition tends to coincide with increases in cortical tracking (Ding & Simon, 2013; Peelle et al., 2013; Vander Ghinst et al., 2016), unless listeners are specifically trained to recover intelligibility through perceptual learning (Kösem et al., 2023; Millman et al., 2015). In other communicative contexts, however, such as unfamiliar accents or second-language listening, enhanced tracking is instead associated with reduced intelligibility, and has been interpreted as a maladaptive compensatory mechanism driven by increased listening effort (Reetzke et al., 2021; Song & Iverson, 2018). Here, we examine another source of variation: emotional prosody. Speakers adjust their speech according to their emotional state, and these acoustic variations shape listener attention, lexical access, and memory. Whether emotional prosody similarly modulates cortical tracking, and how tracking relates to speech recognition across prosodic conditions, remain unknown. To address this, twenty L1 listeners of American English transcribed 420 short, meaningful sentences while 32-channel EEG was recorded. Sentences were produced by a female, American actor in happy, angry, or neutral prosody. Half of the sentences were presented at full spectrum and half were spectrally degraded using a 4-channel noise vocoder. Speech recognition was quantified on a trial-level basis as the proportion of target keywords accurately transcribed. Cortical tracking in the 1–8 Hz range was quantified by measuring phase synchrony (via phase-locking value) and amplitude synchrony (via Pearson correlation) between neural and envelope oscillations. Results were analyzed with linear and logistic mixed effects models. Transcription performance across vocoded trials increased for all three prosodies (ps<.0001), but angry speech was transcribed most accurately (ps<.0001) and showed the fastest and strongest trial-by-trial perceptual adaptation to vocoding (ps<.04). Despite lower recognition accuracy, neutral speech exhibited the strongest phase and amplitude synchrony (ps<.0001). Notably, while phase and amplitude synchrony were enhanced for vocoded relative to full-spectrum speech (ps<.0001), both measures decreased gradually across vocoded trials for happy and angry speech (ps<.003) as transcription performance improved. Direct comparison of neural synchrony and transcription accuracy revealed an inverse relationship between phase synchrony and the recognition of vocoded happy and neutral speech (ps<.004), and a significant but weak positive relationship between amplitude synchrony and the recognition of vocoded angry speech (p=.04). These findings suggest that cortical envelope tracking is quite sensitive to the temporal structure of speech, producing stronger synchrony when envelope fluctuations are smoother over time (neutral prosody) or across frequencies (vocoded speech). Importantly, listeners’ recognition accuracy can also modulate cortical envelope tracking. As listeners adapt and comprehension improves, they may invest fewer resources in encoding the temporal structure of the speech signal. For speech that is already highly intelligible, recognition performance may be further boosted by enhanced temporal tracking.
Topic Areas: Speech Perception, Prosody