Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Speech in motion: neural encoding of speech acoustics and speaker hand kinematics in audiovisual retellings
Poster Session F, Friday, October 2, 2:45 - 4:45 pm, Wangari Maathai
David Hernández-Gutiérrez1,2, Romain Pastureau1, Nicola Molinaro1; 1BCBL, San Sebastián, Spain, 2University of Rochester, NY, USA, 3Ikerbasque, Basque Foundation for Science, Bilbao, Spain
During natural language communication, speech unfolds continuously over time and is tightly accompanied by co-speech body movements. Both signals are temporally synchronized, suggesting that the observation of speaker body motion may influence ongoing auditory speech processing. Although increasing evidence has characterized the kinematic properties and production of body movements, relatively little is known about their impact on listeners and on the neural representation of speech. A recent EEG study using “point-light” speakers found signatures of audiovisual integration in speech envelope reconstruction accuracy and neural temporal dynamics around 150 ms, suggesting that visual processing of speaker motion facilitates speech encoding (Momsen & Coulson, 2025). However, the study did not directly model body kinematics, limiting interpretation to the effect of global body motion and leaving unanswered how listeners encode the motion of specific articulators, such as the hands. Moreover, it remains unknown how listeners encode co-speech body motion from natural speakers. We investigated how speaker hand velocity modulates neural speech encoding using source-localized magnetoencephalography (MEG). First, we created an audiovisual corpus from which the MEG stimuli were selected. Six individuals watched cartoon videos (e.g., The Pink Panther, Tom and Jerry) and were subsequently recorded while retelling the stories, with their body movements tracked using Kinect. These recordings were adapted to different experimental conditions. In the MEG experiment, 30 participants watched and/or listened to 80 cartoon retellings presented in auditory (A), visual (V), and audiovisual (AV) modalities. In addition to a natural condition, in which both the speaker’s body and hands were visible, three further conditions were included: (1) a masked-face condition controlling for visible speech; (2) a point-light condition isolating body- motion cues; and (3) a shuffled point-light condition preserving kinematic properties while disrupting body spatial structure. Participants answered a forced-choice comprehension question after each story to ensure attention. MEG activity was source reconstructed using individual structural MRI scans. Encoding temporal response functions (TRFs) were fitted separately within superior temporal and occipital regions. Neural models included two acoustic speech features together with the velocity of each hand. Results revealed stronger prediction accuracy within the right superior temporal region in the point-light condition relative to both the natural and masked-face conditions. The shuffled point-light condition also showed stronger acoustic encoding than the masked-face condition. TRF weight analyses revealed late effects on acoustic processing, including larger negative weights for A relative to the masked-face AV condition (~260–440 ms), and for the point-light relative to the natural AV condition (~360–480 ms). In left occipital regions, right-hand velocity encoding was stronger for the masked-face condition in the V modality relative to the AV modality. TRF weight analyses further revealed enhanced negative weights for hand-velocity encoding within the AV modality around 250–400 ms. Together, these findings suggest that distinct visual cues dynamically shape speech encoding over time, with some effects emerging specifically when both body motion and visual speech are available, while others arise specifically by natural body motion.
Topic Areas: Multisensory or Sensorimotor Integration, Signed Language and Gesture