Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Zero-Shot LLM-Based CIU Estimation in Aphasia: Reliability and Neuroanatomical Validity
There is a Poster PDF for this presentation, but you must be a current member or registered to attend SNL 2026 to view it. Please go to your Account Home page to register.
Poster D13 in Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai and Lise Girardin
Huiyan Zhang1, Junhua Ding1; 1Institution of Psychology, Chinese Academy of Sciences
Introduction Aphasia is characterized by multi-level impairments in lexical retrieval, syntactic organization, and discourse informativeness (Borovsky et al., 2007). Correct Information Unit (CIU) analysis is widely used to quantify functional communicative impairment in aphasia (Nicholas & Brookshire, 1993), but the procedure requires labor-intensive manual coding and depends heavily on raters. Recent studies have explored the use of large language models (LLMs) for automated discourse assessment in aphasia (Kleiman, 2026; Kurland et al., 2025), building on their strong zero-shot and few-shot language abilities (Brown et al., 2020). However, the reliability and generalizability of LLM-based CIU estimation in aphasia remain unclear. The present study evaluated the behavioral and neural validity of zero-shot CIU estimation using locally deployed LLMs, and further examined the effects of speech genres, aphasia profile, and lesion patterns on model performance. Methods Sixty individuals with post-stroke aphasia completed 10 discourse tasks across three genres: Procedural, Picture Description, and Sequence Pictures. Speech samples were manually transcribed and scored for CIU count by trained raters. Eight locally deployed LLMs were implemented via Ollama and evaluated in a zero-shot setting using a standardized CIU scoring prompt (Nicholas & Brookshire, 1993). Agreement with human scores was assessed using Spearman's rho, ICC(A,1), ICC(C,1), MAE, and mean bias in R. The best-performing model was further applied across different speech genres, aphasia severity, and subtypes (anomic, fluent, non-fluent). Severity was classified using the Western Aphasia Battery Aphasia Quotient (WAB-AQ); analyses were restricted to mild (n = 26) and moderate (n = 15) groups due to insufficient sample sizes in the remaining severity categories. Predictive validity comparison (PVC) analysis (Magnotti et al., 2023) was conducted to compare lesion-behavior relationships derived from human and LLM-based CIU scores while controlling for lesion volume. Results Across eight LLMs evaluated on the Dishes task, qwen2.5:7b demonstrated the strongest agreement with human scoring (rho = 0.615, ICC(A,1) = 0.734, ICC(C,1) = 0.780; Table 1). Cross-genre analyses revealed that Procedural tasks yielded consistently higher ICC values (ICC(A,1) = 0.572, ICC(C,1) = 0.648) than Picture Description(ICC(A,1) = 0.412, ICC(C,1) = 0.540) or Sequence Pictures tasks (ICC(A,1) = 0.364, ICC(C,1) = 0.486), while rho remained moderate across categories (range: 0.58–0.77). Agreement was substantially higher for mild aphasia (rho = 0.664, ICC(C,1) = 0.809) than for moderate aphasia (rho = 0.19, ICC(C,1) = 0.25), indicating that severity of language impairment systematically influenced model accuracy (Figure 1). Aphasia subtype also modulated performance: anomic aphasia yielded the highest agreement (rho = 0.612, ICC(C,1) = 0.769), followed by fluent (rho = 0.394) and non-fluent aphasia (rho = 0.199, ICC(C,1) = 0.334). PVC analysis provided decisive evidence (AIC difference = −193.99) in favor of a shared lesion-behavior relationship between human- and LLM-derived CIU measures. Summary LLMs, particularly qwen2.5:7b, can approximate human CIU scoring in a zero-shot setting. Performance was modulated by genre type, aphasia severity, and subtype, with greatest accuracy in mild and anomic aphasia. LLM-derived scores identified lesion patterns consistent with human annotations, supporting neuroanatomical validity. These findings support zero-shot LLMs as scalable aphasia assessment tools.
Topic Areas: Disorders: Acquired, Language Production