Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Aligning large language models with human word associations for single-word semantic representation.
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
Simon De Deyne1, Sukhai Huang2, Lea Frermann1, Chunhua Liu1; 1University of Melbourne, 2Monash University
Neural encoding and representational similarity studies of language increasingly use LLM embeddings to model the conceptual content of naturalistic passages or stories. LLMs trained on rich linguistic context predict human behavior and neural responses well in these experiments (Hosseini et al., 2024; Antonello, Vaidya, & Huth, 2024). What remains less clear is whether they also capture the meaning of isolated words. Humans tend to agree strongly on the meaning of single words despite the lack of supporting context, as they can draw on perceptual, affective, and experiential knowledge that is represented less prominently in corpora (De Deyne, Navarro, Collell, & Perfors, 2021). Here, word association data might be especially useful for approximating what is represented, as they provide a scalable, transparent, and direct behavioral measure that combines verbal and non-verbal common-sense knowledge. Previous work using association-based semantic networks from Small World of Words (SWOW; De Deyne et al., 2019) has demonstrated this potential to connect behavior and brain data. They not only predict lexical processing and similarity judgments (De Deyne et al., 2019) but also close match neural responses in RSA analyses of single-word fMRI data, in ways that text-based representations cannot (Yang et al., 2024). The current study examined the degree to which fine-tuned LLMs can predict human word associations and whether this fine-tuning improves the prediction of lexico-semantic processing beyond the association task itself. To do so, we fine-tuned Llama-3.1-8B-Instruct on participant-level SWOW responses, using the original task instructions and calibrating generation temperature to match the diversity of human responses. Evaluation was based on held-out SWOW cues and newly collected unpublished data to exclude training-set leakage. Relative to an untuned model, fine-tuning substantially improved the model's ability to reproduce human associative structure. The strongest human association was typically the top-ranked response from the fine-tuned model (median rank = 1 vs. 3 in untuned models), and cue-level response distributions approached the reliability of independent human samples. Importantly, fine-tuning also resulted in a model that acquired characteristic properties of human association behavior: preferences for short, frequent, concrete responses and convergence on the same high-centrality semantic hubs found in human networks. Replicating the procedure in De Deyne et al. (2019), we constructed comparable LLM-derived semantic networks and extracted word-level centrality and pairwise similarity. The fine-tuned model predicted English Lexicon Project lexical decision (Balota et al., 2007) and naming times at levels nearly indistinguishable from human norms, substantially better than the untuned LLM, and produced highly competitive, often best-performing predictions of human semantic similarity and relatedness from limited data. This is important because it is not obvious why a model fine-tuned on word associations would perform better than a baseline model designed to perform well across a variety of language tasks. Fine-tuning on word associations thus has the potential of a broad-scope foundation that brings LLM representations of isolated words closer to how humans represent them and opens pathways to create more bespoke, demographic-aware representations, which we are currently pursuing.
Topic Areas: Meaning: Lexical Semantics, Computational Approaches