Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Multimodality supports neural processing of grammatical operations
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Hatice Zora1,2, Peter Hagoort1,2, Aslı Özyürek1,2; 1Max Planck Institute for Psycholinguistics, 2Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, Radboud University
In contrast to traditional unimodal theories of language, an increasing body of research supports the idea that language functions as a multimodal system. However, previous empirical studies on multimodal language processing have largely focused on lexical-semantic integration, while grammatical operations have received far less attention. Addressing this gap, the present study investigates negation, a fundamental component of human cognition that is grammatically encoded across all languages. Despite its central role, the multimodal processing of negation, especially grammatical negation (e.g., “I did not sleep”), remains poorly understood, and the contributions of gestures and prosodic cues to its neural processing have yet to be examined. This study explores the neurocognitive processing of multimodal negation in Turkish, a typologically distinctive verb-final language, where standard verbal negation is morphologically marked with the post-verbal suffix -mA. Importantly, this suffix shifts prosodic stress onto the verb stem, diverging from Turkish’s default word-final stress pattern. Verbal negation in Turkish is also commonly accompanied by conventional negation gestures, such as backward hand flips, produced on the verb stem before the negation suffix. Because both gestural and prosodic cues precede the morphological negation marker, we investigated whether and how they influence the processing of post-verbal negation using electroencephalography (EEG). The stimuli consisted of 120 short dialogues in Turkish, event-related potential responses (ERPs) were time-locked to the onset of the sentence-final verbs. Prosody and gesture manipulations were applied directly to the verb stems, with prosody either present or absent and gesture either present or absent. The stimuli were presented using digital avatars, and the task of the participants was to listen to the dialogues for comprehension. Data collected from native speakers of Turkish (n = 24) revealed that gestures elicited an N100 effect, likely driven by their perceptual salience. Gestures also attenuated the N400 effect, typically associated with cognitive processing effort, compared to conditions without gestures, indicating that they facilitate negation processing. Prosodic cues further enhanced this N400 reduction, but only when gestures were also present; prosody alone produced no comparable effect. These findings suggest that gestures, and to a lesser extent prosody, reduce the cognitive demands associated with processing morphosyntactic marker of verbal negation. The results support the Maximizing Online Processing principle, according to which efficiency is enhanced by choosing and organizing linguistic forms in ways that allow the earliest possible access to the syntactic and semantic representations. In this case, processing efficiency is enhanced through the early availability of gestural and prosodic cues, despite the late arrival of morphological negation marking. Overall, the study provides the first neurophysiological evidence that multimodal integration contributes not only to lexical-semantic processing, as previously demonstrated, but also to core grammatical operations in language.
Topic Areas: Multisensory or Sensorimotor Integration, Signed Language and Gesture