Search Abstracts | Symposia | Slide Sessions | Poster Sessions
PsychLing Benchmark: Psycholinguistic metrics for Human-Model Corpus Comparison in Children and Adults
Poster Session F, Friday, October 2, 2:45 - 4:45 pm, Wangari Maathai
Hanna Woloszyn1, Cosimo Iaia2, Benjamin Gagl1; 1University of Cologne, 2Goethe University Frankfurt
Language is complex, and linguistic skills vary widely across individuals and development, raising the question of how to measure individual differences and longitudinal change in language output. We combine 19 metrics capturing central psycholinguistic dimensions: textual, lexical, syntactic, part-of-speech, and semantic characteristics into the PsychLing Benchmark, designed to compare similarities and differences across human- and model-generated corpora. As a case study, we compare German picture-story descriptions produced by children of different ages (i.e., longitudinal data), adults, and a vision-language model (GPT-4o). For each human text, we simulated LLM texts using prompting strategies that incorporated increasing amounts of participant-specific information: (i) age only, (ii) metadata, (iii) metadata plus same-story examples from the same child, and (iv) metadata plus examples from all eight picture stories. Across most measures, LLM-generated texts differed significantly from human texts and rarely reproduced individual differences or longitudinal change. However, the results followed the expected pattern: adult language showed the strongest, though still limited, alignment, and child-language simulations improved when we provided more participant-specific information. The PsychLing Benchmark thus provides a theoretically motivated, interpretable set of metrics for comparing human- and model-generated language, supporting targeted model evaluation and development, extending current available NLP benchmarks.
Topic Areas: Language Development/Acquisition,