Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Does audiovisual enhancement of speech tracking rely on meaning?
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Kira Benazzouz1, Seana Coulson1; 1UC San Diego
A large body of research has demonstrated that co-speech gestures influence numerous aspects of spoken language processing. Speech prosody is one such aspect; previous studies have shown that congruent co-speech gestures can enhance the brain’s tracking of the speech envelope. This may suggest that visual information from co-speech gestures allows the brain to track speech prosody more efficiently. However, it remains unclear whether this effect relies on gestures contributing to the meaning in speech in a way that improves prosody tracking rather than exhibiting a direct influence. The present study investigated the influences of point-light depictions of natural co-speech gestures and audio-vocoding on cortical speech envelope tracking. Audio vocoding was used to degrade the intelligibility of the speech and obscure semantic information while retaining the speech envelope of the original speech that participants heard. Point-light gesture depictions were employed to retain fundamental aspects of the body’s kinematics while obscuring other cues from the speaker’s face, body, and environment. Together, these manipulations will allow us to determine whether point-light depictions of co-speech gestures influence cortical speech envelope tracking and whether this effect emerges when no semantic information is available from the speech. Accordingly, we recorded EEG while participants viewed short clips extracted from the Trinity Speech-Gesture dataset. This dataset includes videos of an actor responding to prompts naturally while gesturing freely. The videos were presented in four different conditions: audio-only, vocoded audio-only, audiovisual, and vocoded audiovisual. The audio-only conditions included only the speech from the original clips while the audiovisual conditions included both the speech and gestures. After each video was presented, participants were presented with a probe word and asked to indicate with a key press whether the word was said in the preceding video. We then used a multivariate temporal response function (mTRF) neural decoders to compare the fidelity of neural speech tracking in the four different conditions. The mTRF decoder models were used to reconstruct the speech envelopes given EEG data recorded as the participant viewed the clips. Pearson correlations between the reconstructed speech envelopes and the true speech envelopes are referred to as decoder scores. Decoder scores demonstrate the relationship between the EEG signal and the speech envelopes, elucidating the strength of the brain’s tracking of the speech envelope in each condition. Preliminary analyses of our partially collected data demonstrate that the decoder scores were significantly above chance in all conditions. However, no significant effects of vocoding or co-speech gestures were observed. Our results suggest that the brain reliably tracked speech envelopes across conditions, but the strength of the tracking was not influenced by either the intelligibility of the speech or by whether the speech was accompanied by gestures. Further data collection and analyses are necessary to clarify these findings.
Topic Areas: Multisensory or Sensorimotor Integration,