Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Late emergence of cross-modal semantic generalization between words and images in a sequence learning paradigm
Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai
Nicolas Piron1, Théo Desbordes1, Itsaso Olasagasti1, Sophie Schwartz1, Nina Kazanina1; 1University of Geneva
A central question in cognitive neuroscience is how and under what conditions the brain arrives at representations that are independent of input modality. Neuroimaging studies have shown that different stimulus modalities activate common brain areas and shared representations of concepts (Meyer et al., 2010; Simanova et al., 2010), pointing to the existence of amodal semantic representations considered to be supported by a conceptual hub in the anterior temporal lobe (Lambon Ralph et al., 2017). Yet the temporal dynamics of when such representations emerge remain poorly understood. If amodal semantic representations exist, neural activity patterns evoked by words and images representing the same concepts should be mutually decodable. By training classifiers on MEG data from one modality (e.g. images) and testing them across all timepoints of the other (e.g. words), one can estimate when a shared representational code is first reached, whether it is maintained or reactivated over time, and in which direction this temporal transfer applies across modalities. Previous work by Dirani & Pylkkänen (2023) demonstrated that classifiers trained to distinguish animals from tools in one modality generalized to the other, as early as ~150 ms during picture naming and word reading. Earlier word representations generalized to later image representations and vice versa, suggesting faster access to amodal semantic content for words than for images. Here we extend this approach to a broader and more heterogeneous semantic space using a sequence learning paradigm, in which participants are exposed to words and images incidentally rather than performing explicit naming or reading tasks. Twenty-five participants learned sequences of written words and sequences of images, where items were drawn from six semantic categories (characters, landscapes, shapes, colors, animals, body parts), while MEG was recorded. Using a one-versus-rest approach, Lasso regression classifiers were trained to discriminate each semantic category from all others in one modality and tested on the other for all pairs of timepoints, yielding a time-resolved measure of cross-modal generalization. Bidirectional cross-modal generalization was observed across all six categories, demonstrating that modality-independent representations extend beyond coarse animacy distinctions to a broader and more heterogeneous semantic space. However, unlike in Dirani & Pylkkänen (2023), generalization emerged only after ~500 ms and showed the opposite asymmetry: earlier image (~500 ms) representations generalized to later word representations (~600 ms), and vice versa. This latency falls well outside the window of early perceptual and semantic categorization, which typically occurs within 100–400 ms, suggesting that in this paradigm, convergence onto shared representational content is driven by a late, likely task-dependent process rather than automatic early semantic access. Our findings suggest that convergence onto a common semantic code is not an automatic consequence of stimulus processing but rather depends on task demands and the nature/versatility of semantic categories involved. More generally, access to an amodal representation is a flexible, context-sensitive process, with implications for models of semantic memory that assume fixed, task-independent amodal representations.
Topic Areas: Multisensory or Sensorimotor Integration,