Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Decoding lexical semantic information in temporoparietal cortices
Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai
Joon Hei Lee1, Tianhao Lei1, Joshua Rosenow1, Jason Hsieh1, Joshua Glaser1, Marc Slutzky1; 1Northwestern University
Brain-computer interfaces (BCIs) hold promise for restoring communication to patients with speech deficits caused by neurological impairments. Existing speech BCIs rely almost exclusively on neural signals from frontal cortical areas tied to phonological and articulatory output. This inherently excludes a significant population of individuals with deficits caused by frontal lobe damage, such as expressive aphasia. Leveraging semantic information is a compelling alternative approach, as semantic processing is known to involve distributed activation beyond the frontal lobe—particularly in temporal and parietal cortex—but directly decoding semantics from neural activity in these regions alone remains underexplored. We therefore investigated whether lexical semantic content can be decoded from temporoparietal cortical areas during speech production, and whether such representations are modality-invariant and linguistic rather than perceptual in nature. Electrocorticography (ECoG) and stereo-electroencephalography (sEEG) signals were recorded from 12 patients undergoing monitoring for intractable epilepsy, during a picture naming task; a subset (n=7) additionally completed an auditory description naming task. We extracted high-gamma activity (70-200 Hz) and selected a subset of features using neighborhood components analysis. We then input these features into support vector machine classifiers to decode semantic category (6 classes) and word identity (15 classes; picture naming only). To understand the spatiotemporal dynamics of these semantic representations in the brain, we performed decoding across time using 500-ms sliding windows of data as inputs and used data from individual, standard anatomical regions of interest (ROIs). We also regressed neural activity during picture naming onto language (GloVe, ConceptNet) and visual deep learning (DINOv2, SimCLR) embedding spaces to characterize the representational content of the decoded signals. Above-chance decoding accuracy using temporoparietal signals was achieved for all 12 participants in the picture naming task and all 7 participants in auditory description naming. Peak accuracy occurred roughly 500-700 ms after stimulus onset for picture naming, and prior to speech onset in both tasks. ROI analyses identified several shared loci of decodable semantic content across both tasks, including anterior and posterior ventral temporal lobe. Neural activity was also better accounted for by language embedding spaces than visual embedding spaces, suggesting that our decoding results stem specifically from linguistic representations and not simply visual processing of image stimuli. These results demonstrate that temporoparietal cortical areas, particularly ventral temporal cortex, contain decodable lexical semantic representations that converge across modalities. These areas may serve as potential substrates for lexical-semantics-based speech BCIs to help people with aphasia.
Topic Areas: Meaning: Lexical Semantics,