Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Toward Automated Narrative Macro-Structure Scoring: A Multi-LLM Reasoning Protocol
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Smadar Patael5, Daniel Peretz1, Nili Erlich2, Yarrit Zvi3, Noa Yaakobi4; 1Tel Aviv University, Israel
Introduction. Narrative macro-structure, organizing events into a coherent episode, is reliably disrupted after acquired brain injury, yet quantifying it depends on expert manual scoring that is slow and prone to rater disagreement. We are developing a novel architectural multi-model in which several large language models score macro-structure via explicit reasoning, justify each score with textual evidence, and have their independent judgments reconciled into a consensus, with no training corpus. We ask whether it can score narratives faithfully, and whether it reproduces the disagreement pattern human scorers themselves show - a signature of construct ambiguity rather than model error. Our goal is a validated, reliable LLM-based system for narrative analysis that supports the assessment of clinical populations, particularly children with acquired brain injury. Methods. The protocol was developed from 240 Hebrew narratives elicited from 60 typically developing children using four picture prompts. Each is scored on seven macro-structure components - character, setting, initiating event, internal response, plan, action, consequence - on a 0–3 scale anchored on causal-temporal coherence. In calibration, 30 narratives were scored independently by four judges - three language models (GPT, Gemini, Claude) and a trained human rater - each returning, per component, a score, a justification with quotes, and a confidence rating. An automated check verified that every cited quote genuinely appeared and matched its score. A discourse adjudicator resolved disagreements and folded the resulting precedents into an expanded rubric. In the scoring stage, the three models scored the remaining 210 narratives; low-agreement, low-confidence, or flagged cases were escalated to the adjudicator. A random 20% are being independently blind-scored. Results. Preliminary results follow the protocol's stages. First, grounding was reliable - the automated check confirmed genuine, score-consistent evidence in over 99% of judgments: models anchored scores in the narrative rather than fabricating evidence. Second, calibration revealed a clear difficulty gradient: agreement with the human rater (quadratic-weighted kappa) was highest for initiating event - one model reaching κ = 1.00 - and lowest for plan, where all three models fell below the conventional 0.70 threshold. Third, this ordering recurred in the scoring stage: scores cleared all automated acceptance criteria at rates from roughly 70% to mid-80%, again, the lowest for the plan. The gradient appeared across two independent measures, not as an artifact of either. Discussion. These findings raise three points. First, verification suggests the models ground their scores in the text rather than fabricating them - a prerequisite for trusting automated scores. Second, and tellingly, the difficulty gradient tracks human scoring: plan, the hardest component, is also where human inter-rater agreement is lowest (Jones et al., 2019). A protocol whose uncertainty mirrors where humans disagree may be sensitive to genuine construct ambiguity rather than failing arbitrarily - an early sign of construct validity. Third, the multi-model design points to a direction not yet pursued: comparing models to characterize their model-specific biases. This work is preliminary, validated only on typically developing narratives; applying the protocol to the collected narratives of acquired brain injury comes next.
Topic Areas: Meaning: Discourse and Pragmatics, Methods