Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Better Language Models Better Model the N400, but not Reading Time
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
James Michaelov1, Benjamin Bergen2; 1Massachusetts Institute of Technology, 2University of California San Diego
The probability of a word in context, as captured by statistics-driven computational language models, is predictive of both behavioral and neural measures of human language processing (McDonald and Shillcock, 2003; Frank et al., 2015). In contrast to previous research (e.g., Goodkind and Bicknell, 2018), recent work suggests that language models can become too good at next-word prediction to model reading time (e.g., Kuribayashi et al., 2021). However, it is unknown whether this decoupling is true of reading time only, or whether it is intrinsic to online measures of comprehension more generally, reflecting something more fundamental. To address this question, we turn to the N400 (Kutas and Hillyard, 1980). We calculate word surprisal (negative log-probability) using Pythia (Biderman et al., 2023), a collection of transformer language models ranging in size (14 million to 12 billion parameters) and training data (0 to 300 billion tokens); and use these surprisals to predict N400 amplitude data from 9 previously-published studies (Federmeier et al., 2007; Nieuwland et al., 2018; Wlotko and Federmeier, 2012; Hubbard et al., 2019; Lago et al., 2019; Ryskin et al., 2021; Szewczyk et al., 2022; Szewczyk and Federmeier, 2022; Michaelov et al., 2024) and reading time from 4 previously-published datasets (Kennedy et al., 2003; Smith and Levy, 2013; Luke and Christianson, 2018; Futrell et al., 2021) using linear mixed-effects regressions. We analyze the fit (AIC) of these regressions to the data, testing how it varies depending on language model properties. We replicate and extend previous work (Oh and Schuler, 2023), finding that during the first 2 billion tokens of training, models trained on more data calculate surprisals that better fit reading time (χ² (1)=10.21, p = 0.0299), but after this point, models trained on more data perform worse (χ² (1) = 10.19, p = 0.0299) and larger models perform worse (χ² (1) = 11.20, p = 0.0254). We observe a very different pattern for N400 amplitude: during the first 2 billion tokens, language model surprisal provides a consistently poor fit to the N400 data with no change, but after this point, larger models perform better (χ² (1) = 24.31, p < 0.0001), models trained on more data perform better (χ² (1) = 24.02, p = 0.0001), and this latter effect is increased for larger models (χ² (1) = 8.87, p = 0.0439). We carry out further comparisons with natural language processing benchmarks, finding that the first 2 billion tokens of training is when the models most improve at simple next-word prediction (see also Michaelov et al., 2025), while after this, they increasingly improve at generating semantically coherent text. This suggests that reading time may reflect a sensitivity to more surface-level statistics, while the N400 may more strongly reflect semantic aspects of language processing (Kutas and Federmeier, 2011; Van Petten and Luka, 2012). Our work also highlights that while language models can be useful for capturing statistical probability, care should be taken in interpreting results—‘psychometric predictive power’ (Wilcox et al., 2020) does not necessarily generalize across metrics of human language processing.
Topic Areas: Computational Approaches,