Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Cortical Dynamics of Speech-Visual Semantic Integration
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Shristi Baral1, Riitta Salmelin1, Marijn van Vliet1; 1Aalto university
The brain receives information about the world from multiple sensory pathways, but meaningful understanding depends on recognizing when different inputs refer to the same object, action, or event. Signals such as the sight, tactile texture of an object, sound they make or sound of their name are physically distinct, yet they must be combined into a coherent representation. Here, we focus on the mapping between linguistic-audio and non-linguistic visual stimuli. Given that one is perceived through eyes and one through ears, our study aims to gain insight as to when and where the integration occurs. Neuroscience literature points to three likely candidate areas that participate in integration of visual-object and spoken-language information: the anterior temporal lobe (ATL) the temporal-parietal-occipital junction (TPOJ), including the angular gyrus (AG), each with evidence of flexible pathways and diverse networks to support the integration. In an MEG experiment, we simultaneously presented both spoken and visual stimuli that either describe the same scene or conflicting scenes. By tracking when and where the mismatch effect becomes first observable in the brain, we aimed to pinpoint where the pathways meet. The participants were 30 healthy, right-handed native Finnish speakers aged 18–40. They were presented with line-drawn illustrations of a subject interacting with an object, paired with three-word spoken sentences describing such an illustration. The sentence and illustration were presented simultaneously. Across trials, the object of the spoken sentence, i.e., the last word was manipulated so that it was either congruent, semantically related (same semantic category), or incongruent with the illustration. Visual-only and speech-only conditions were also included to estimate unimodal responses and to compare them with the multimodal speech-visual responses. Based on preliminary analysis, the cluster permutation test revealed significantly stronger activation (p<0.05) in the incongruent than congruent condition in the left posterior temporal lobe near TPOJ area at 300-700 ms. When spoken sentences and illustrations were presented together, responses were sub-additive relative to the summed unimodal conditions, suggesting that multimodal context facilitates semantic processing by reducing the neural response required to construct meaning. These findings will be discussed in relation to theories of semantic processing and multimodal semantic integration, i.e., hub-and-spokes model, convergence zone theory, and grounded cognition theory. With the preliminary analysis indicating involvement of the left posterior temporal and TPOJ regions in speech–visual integration, we may have evidence in support of grounded cognition and/or convergence zone theory of semantic integration.
Topic Areas: Meaning: Lexical Semantics,