Search Abstracts | Symposia | Slide Sessions | Poster Sessions
rERP Evidence for the Role of Kinesthetic and Verbal Working Memory in Iconic Gesture Processing During Discourse Comprehension
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Zoe Ezzes1,2, Kira Benazzouz1, Seana Coulson1; 1University of California, San Diego, 2San Diego State University
Iconic gestures, or manual movements temporally aligned and semantically related to the content of co-occurring speech, have been demonstrated to impact discourse comprehension, particularly for visuospatial properties of discourse referents. Prior research supports speech-gesture congruency effects, with facilitation of speech comprehension indexed by faster reaction times, more correct responses, and less negative N400. However, not all individuals appear to be impacted the same. Individual variability in kinesthetic working memory (KWM) ability has been found to be positively correlated with speech-gesture congruency effects as measured by reaction time differences. Specifically, individuals with higher KWM tend to benefit more from the presence of congruent gestures, while individuals with lower KWM tend to experience greater degrees of interference from incongruent, and even weakly congruent, gestures. Prior studies have also examined the role of verbal working memory (VWM) ability in the on-line integration of conceptual information across simultaneously presented modalities. While some prior work indicates that speech-gesture integration makes minimal demands on VWM, one study found that manipulating VWM load resulted in qualitatively different speech-gesture incongruity effects, with lower VWM load resulting in a more negative N400 in a fronto-central distribution and higher VWM load resulting in a more positive P3a frontal distribution. Given such findings, the impact of individual differences in VWM ability on co-speech iconic gesture integration warrants further investigation. In the present study, EEG was recorded as healthy undergraduate participants viewed multimodal video clips containing spoken discourse coupled with either congruent or incongruent iconic gestures. Each clip was preceded by a short title (consistent across congruent and incongruent conditions) and followed by a picture probe that was semantically related to the spoken content. Participants also underwent behavioral testing to obtain individual scores for KWM and VWM abilities. We plan to compute rERPs to assess the degree to which individual differences in KWM and VWM affect real-time neural responses to word onsets during multimodal discourse comprehension and to subsequent picture probes. Overlap correction will be applied due to the temporal proximity of presented words in discourse. Fixed effects will include KWM score, VWM score, speech-gesture congruency, and the interactions between congruency and each working memory score. Given prior work suggesting that sensitivity to gesture congruency is related to KWM capacity, we predict interaction effects between KWM ability and speech-gesture congruency. Further, with prior work indicating lesser demands of speech-gesture integration on VWM, we predict VWM effects to be independent of speech-gesture congruency, as would be suggested by additive effects. Observation of predicted effects would provide support for the spatio-motoric hypothesis relating KWM ability to speech-gesture congruency effects and help clarify the import of VWM in multimodal discourse comprehension. Results will help characterize how individual variability in KWM and VWM is manifested in the electrophysiological response to speech accompanied by congruent and incongruent gestures.
Topic Areas: Signed Language and Gesture, Multisensory or Sensorimotor Integration