Search Abstracts | Symposia | Slide Sessions | Poster Sessions
The Kinematic Texture of Meaning: Biological Motion Supports Semantic but Not Surprisal Tracking
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Haoyin Xu1, Jacob P Momsen2, Seana Coulson1; 1Department of Cognitive Science, University of California, San Diego, 2Yale School of Medicine
Successful speech comprehension is inherently multimodal: listeners integrate spoken language with visible movements of the face, hands, and body. While co-speech gesture is known to support comprehension, most work examines it alongside speech and visible talker form. Here, we asked whether the bare kinematic trace of gesture—isolated from acoustic speech, facial detail, articulation, and realistic body form—carries information about the evolving meaning of discourse. Building on Broderick et al. (2018), who showed that brain activity tracks word-by-word semantic dissimilarity as natural speech unfolds, we tested whether similar discourse-level semantic changes could be detected when listeners heard no speech, but saw the silent biological motion of a speaker’s gestures. Participants’ EEG was recorded as they viewed short discourse excerpts in four formats: silent point-light biological motion (Vo), audio-only speech (Ao), congruent audiovisual point-light talkers (AVc), and incongruent audiovisual point-light talkers in which speech from one excerpt was paired with gesture from another (AVi). Clips were drawn from the Trinity Speech–Gesture Database (Ferstl & McDonnell, 2018) and rendered as point-light talkers, preserving the timing and trajectories of the speaker’s head, trunk, arms, and legs while removing facial, articulatory, and realistic body-form cues. We used backward temporal response function (TRF) modeling to ask whether the EEG could be used to reconstruct two word-level time series derived from the spoken transcripts: lexical surprisal and semantic dissimilarity. Lexical surprisal indexed how unexpected each word was given the preceding words; semantic dissimilarity indexed how much each word changed the meaning built from the preceding discourse context. Reconstruction accuracy was quantified as the correlation between the EEG-reconstructed time series and the original transcript-derived time series, and significance was assessed with permutation tests. Lexical surprisal reconstruction in the silent biological-motion condition did not reliably exceed chance (Vo mean = 0.022, SE = 0.0075, p = 0.08), suggesting that point-light gesture alone was insufficient to support sensitivity to word-by-word probability structure. Audio-only performance was higher than visual-only performance but did not differ reliably from either audiovisual condition (Ao mean = 0.069, SE = 0.013; AVc mean = 0.066, SE = 0.0071; AVi mean = 0.059, SE = 0.0087), indicating that surprisal tracking was driven primarily by speech. Semantic dissimilarity showed a broader pattern, with reconstruction accuracy exceeding chance across all conditions (Ao mean = 0.065, FDR p = 0.0016; Vo mean = 0.070, FDR p < 0.001; AVc mean = 0.094, FDR p < 0.001; AVi mean = 0.083, FDR p < 0.001). Notably, above-chance reconstruction in the silent biological-motion condition indicates that point-light gesture kinematics alone carried information about the evolving conceptual representation of the discourse. Accuracy was numerically highest in the congruent audiovisual condition and significantly greater than in silent biological motion (p = 0.023), suggesting that semantic tracking was strongest when speech was accompanied by congruent biological motion. Together, these findings suggest that co-speech gesture kinematics contribute to broader conceptual aspects of comprehension, while fine-grained lexical prediction remains predominantly supported by speech.
Topic Areas: Multisensory or Sensorimotor Integration, Meaning: Discourse and Pragmatics