Search Abstracts | Symposia | Slide Sessions | Poster Sessions
A lateral temporo-parietal network for multimodal combinatorial semantic processing
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Gina Humphreys1, Matt Lambon Ralph1; 1University of Cambridge
Successful time-extended semantic cognition requires the brain to integrate information continuously to form a coherent mental model of the world. The brain can build up and update a dynamic semantic representation (a “semantic gestalt”) as each new word or non-verbal stimulus is processed. Here we investigate the neural basis for multimodal combinatorial semantic processing. The lateral temporo-parietal network, including the superior temporal sulcus (STS), extending from the anterior temporal lobe (ATL) to the temporo-parietal junction (TPJ) (henceforth the ATL-STS-TPJ network), may play a key role in combinatorial semantic processing, particularly in verbal tasks. For instance, the ATL-STS-TPJ network shows increased activation for meaningful sentences vs. word lists, meaningful sentences vs. non-meaningful sentences, or when reading a narrative that is congruent with a prior context vs. unrelated in meaning (Branzi, Humphreys et al., 2020; Humphries et al., 2006; Humphries et al., 2007; Vandenberghe et al., 2002; 2004). Combinatorial semantic processing occurs not only in verbal tasks but also in the non-verbal domain, such as processing the evolving visual world around us. However, there has been relatively little research investigating whether verbal and non-verbal tasks engage a common or distinct system. Furthermore, outside the domain of semantic cognition, a very similar ATL-STS-TPJ network is considered to be part of a network specialised for Theory of Mind (ToM) processing, that is, the ability to understand that others have different thoughts and feelings from oneself (Carrington & Bailey, 2007; Schurz et al., 2021). This network alignment could be explained by: 1) a ToM confound in existing language studies, since the majority of experiments include social stimuli (e.g. descriptions of a human performing an action); or 2) given that ToM tasks also involve computing a coherent time-extended semantic narrative, it is possible that these findings could be integrated into a more general “semantic gestalt” model. In two fMRI studies, we investigated: 1) the extent to which verbal and non-verbal combinatorial semantic processing relies on a common or distinct neural system, and 2) the extent to which the ATL-STS-TPJ network is engaged by combinatorial semantic processing, or whether it is specialised for ToM tasks. In Study 1, using a 2 × 2 design, participants were presented with semantically coherent vs. scrambled verbal narratives and non-verbal cartoon strips. The results showed a common bilateral ATL-STS-TPJ network for combinatorial semantic integration, with the entirety of the network showing stronger activation for coherent vs. incoherent items in both the verbal and non-verbal tasks. In Study 2, participants completed a False Belief ToM task (Deen et al., 2015; Dodell-Feder et al. 2011). As expected, activation was stronger for ToM > non-ToM items in the ATL-STS-TPJ network. Critically, however, the ToM effect was eliminated when controlling for important semantic and linguistic properties known to influence the ease of semantic integration. Together, these results provide evidence in favour of a lateral temporo-parietal network for multimodal combinatorial semantics.
Topic Areas: Syntax and Combinatorial Semantics,