Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Sustained neural responses to the visual component of audiovisual speech in human superior temporal cortex
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Michael Beauchamp1, Emma Karn1, Yue Zhang2, John Magnotti1; 1University of Pennsylvania, 2Baylor College of Medicine
The superior temporal cortex is a key brain locus for human speech perception. One model is that the superior temporal gyrus (STG) processes auditory speech (A) while the superior temporal sulcus (STS) processes the face and mouth movements comprising visual speech (V) and integrates information from the face and voice during perception of audiovisual speech (AV) (Zhang et al., Journal of Neuroscience, 2025). To test this model, we presented A, V and AV speech to epilepsy patients implanted with stereotactic EEG (sEEG) electrode shafts. In the study of Zhang et al., only single words were presented, leaving unresolved the question of whether multisensory effects were primarily driven by transient responses to word onset. To answer this question, in the current study, the stimulus material consisted of longer narratives (stand-up comedy videos). The narratives were broken into 5 second blocks, presented in A, V and AV format, separated by 0.5 second inter-block intervals. The amplitude of z-scored broadband high-frequency activity (BHA, 70-150 Hz) was measured in an “early” window (0 - 0.5 s after block onset) and “late” window (0.5 - 5 s) to dissociate transient and sustained responses. Across 13 participants, 81 STS and STG electrodes were identified as "bimodal" (significant response in the early window to both A and V speech). All electrodes showed a sharp initial rise in BHA, peaking at 250 ms after block onset. One group of electrodes showed a “visual sustained” response pattern, responding to visual speech in the early window (mean response amplitude, z = 0.11) and the late window (0.07). Another group of electrodes showed a “visual transient” response pattern, with significant responses to visual speech in the early (0.09) but not the late window (0.01). The response to AV speech was examined; this analysis was unbiased because the AV response was not used to select or classify electrodes. For “sustained” electrodes, there was a significant difference between AV and A responses, with AV > A in both early and late analysis windows (early response peak amplitude: AV = 0.45; A = 0.32; late response, AV = 0.30; A = 0.25.) For “transient” electrodes, there was not a significant difference between AV and A responses in either window. Peak AV response in the early window was higher in "transient" than "sustained" electrodes, so the lack of a sustained response was not attributable to an overall lower level of activity. In the STS, 83% (35 / 42) of electrodes showed a “visual sustained” response pattern, while in STG, 64% (24 / 38) of electrodes showed a “visual sustained” response patterns (p = 0.04 from chi-square). These results are compatible with the idea that STS is especially important for integrating A and V speech: "visual sustained" responses were more prevalent in the STS than the STG, and electrodes with a “visual sustained” response pattern showed a significantly larger response to AV speech than to A or V speech presented on its own, a neural signature of multisensory integration.
Topic Areas: Speech Perception,