Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Visualizing Language in Aphasia: A Language-to-Image Framework for Transmodal Assessment
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
Dinghan Dong1, Junhua Ding1; 1Institution of Psychology, Chinese Academy of Sciences
Introduction Aphasia after stroke is associated with deficits in lexical retrieval, syntactic processing, and discourse-level semantic integration, which significantly impair communication and everyday functional recovery (Lee et al., 2015; Borovsky et al., 2007). Current clinical assessments mainly depend on clinicians’ overall judgments and linguistic analyses of spontaneous speech, which are often time-consuming, susceptible to rater variability, and limited in capturing subtle semantic and communicative impairments (Kertesz, 2006; Basilakos et al., 2014). Recent developments in language-to-image generative AI models have created new possibilities for converting textual semantic information into visual representations (Mitrevska et al., 2025; Doğan et al., 2025). Such approaches may offer a more efficient supplementary tool for aphasia assessment and provide information beyond conventional linguistic measures. Therefore, the present study developed a language-to-image framework to investigate a novel transmodal approach to aphasia assessment. Method We recruited 60 individuals with aphasia who completed the Cat in Tree picture description task (Nicholas & Brookshire, 1993). Spoken discourse samples were audio-recorded and transcribed for subsequent analysis. A locally deployed LLMs was developed to construct a language-to-image pipeline. First, the model extracted core semantic information from patients’ narratives, including subjects, actions/states, environmental context, and omitted elements relative to the target picture stimulus (Ollama, gpt-oss:20b). These structured semantic features were subsequently transformed into standardized prompts for image generation. The prompts, together with the original reference image, were then input into the FLUX Kontext Dev image-generation model within ComfyUI to produce reconstructed images intended to reflect patients’ semantic representations. Subjective image evaluation focused on four dimensions: comprehensibility, semantic richness, consistency with reference, and semantic coherence. Five trained raters independently evaluated all generated images using 7-point Likert scales. Forward stepwise regression analyses were conducted to determine whether subjective image dimensions predicted PCA-derived language components. In addition, preliminary lesion–behavior analyses were performed using the multivariate sparse canonical correlation analysis (SCCAN) algorithm to investigate the neural correlates associated with subjective image quality. Results All four subjective image dimensions demonstrated good-to-excellent inter-rater reliability (ICC3k = .777–.953) and were highly correlated with one another (r = .78–.94, all p < .001). In the forward stepwise regression analyses, comprehensibility significantly predicted the PCA-derived language production and fluency component (beta = 1.17, p = 0.006). Semantic coherence significantly predicted both the language production and fluency component (beta = -1.08, p = 0.008) and the semantic processing component (beta = -1.01, p = 0.02; Table 1). Preliminary lesion–behavior analyses identified associated lesion distributions in left frontal–parietal language-related regions for comprehensibility and semantic coherence ratings, whereas the remaining two subjective dimensions did not demonstrate reliable lesion–behavior relationships (Figure 1). Conclusion These findings support the feasibility of locally deployed language-to-image pipelines for quantifying semantic and discourse-level impairments in aphasia. Subjective evaluations of the reconstructed images were reliably associated with both language deficits and lesions in language-related brain regions. Overall, the results suggest that AI-generated image quality may serve as a novel visualization-based indicator of language dysfunction in aphasia.
Topic Areas: Disorders: Acquired, Language Production