Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Deciphering the "Black Box": Mapping BERT-Derived Semantic Embeddings to Behavioral and Lesion Correlates in Aphasia
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Hening Ji1, Junhua Ding1; 1Institute of Psychology, Chinese Academy of Sciences
Introduction Connected speech provides a clinically informative window into aphasia, reflecting lexical retrieval, discourse organization, and communicative informativeness of speech impairments (Nicholas & Brookshire, 1993; Bryant et al., 2016). Large language models (LLM) and AI are developing rapidly. Recent studies using LLM have successfully quantified the impairments of connected speech in aphasia (Cong et al., 2024; Mirheidari et al., 2024). Even though LLM evaluation is efficient and reliable, its high dimensionality makes it hard to explain and still lacks clinical and neural implications. This study examined whether semantic embeddings of aphasic picture descriptions can be reduced to interpretable components and linked to established behavioral language dimensions and brain lesion patterns. Methods Speech samples were obtained from 80 individuals with aphasia during the picture-description task ‘Picnic Scene’. For each transcript, 768-dimensional BERT embeddings were extracted by mean pooling across token-level hidden states (Devlin et al., 2019; Rogers et al., 2020). Principal component analysis (PCA) with varimax rotation was applied to the participant-by-embedding matrix, and the number of retained components was determined by Horn’s parallel analysis. Seventeen classic language measures, including clinical scores (e.g., WAB-AQ), word indices (e.g., repetition, comprehension) and discourse indices (e.g., word count, correct information units), were similarly reduced to three behavioral components using PCA. Repeated 10-fold cross-validated general linear models tested whether BERT-derived components can successfully predict these behavioral components as a whole, with significance assessed using 1,000-iteration permutation tests. Stepwise regression analyses were then used to characterize the behavioral profile of each individual component. Behaviorally meaningful components were further entered into sparse canonical correlation analysis for lesion-symptom mapping (SCCAN-LSM) to reveal their neural basis. Results Seventeen classic behavioral measures yielded a stable three-component structure: Production & Fluency, Phonology & Repetition, and Semantic Processing. BERT model yielded 13 semantic components. These semantic components reliably predicted all three behavioral components (cross-validated observed–predicted R2s = .854, .388, and .509; ps = .001). Stepwise regression showed that distinct subsets of semantic components contributed to each behavioral dimension (ps < .001, Table 1). Eight of thirteen semantic components contributed to the Production & Fluency ability. Three semantic components additionally contributed to Phonology and Semantic abilities. Exploratory SCCAN-LSM identified significant lesion-based associations for a subset of behaviorally meaningful semantic components. For example, RC4, which contributed to Production & Fluency in the behavioral regression model, showed significant lesion-based prediction (cross-validation correlation = .26, p = .04; Figure 1). The associated regions included brain language regions, such as inferior frontal gyrus, angular gyrus and postcentral gyrus. Conclusions BERT-derived embeddings of aphasic connected speech can reliably predict established behavioral language dimensions. Behavioral regression and lesion-symptom mapping analyses suggest that some semantic components are interpretable, capturing behavioral and lesion-relevant variations of aphasia. These findings enhance the understanding of large language model and may help the clinical application of LLMs on stroke aphasia.
Topic Areas: Disorders: Acquired, Language Production