Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Neural decoding across core language dimensions during naturalistic speech comprehension: from word form to pragmatics
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Riccardo Venturini1,2, Cosimo Iaia2,3, Paolo Canal1, Federico Frau1, Luca Bischetti1, Marco Tettamanti4, Valentina Bambini1, Alessandro Tavano2,3; 1Laboratory of Neurolinguistics and Experimental Pragmatics (NEPLab), University School for Advanced Studies IUSS, Pavia, Italy, 2Institute of Psychology, Goethe University Frankfurt, Frankfurt am Main, Germany, 3Cooperative Brain Imaging Center (COBIC), Goethe University Frankfurt, Frankfurt am Main, Germany, 4Department of Psychology, University of Milano-Bicocca, Milan, Italy
Recently, the broader adoption of machine learning approaches in the cognitive neuroscience of language has enabled researchers to use more ecologically valid paradigms, such as naturalistic speech comprehension. In this study, we aimed to decode the information content of the electroencephalography (EEG) signal to predict theoretically defined core dimensions of language, in a conceptual replication and extension of previous work[1] by examining higher-level language representations such as conceptual semantic and pragmatic features. In line with the hierarchical dynamic coding (HDC) hypothesis[1], these higher-order representations are expected to exhibit more temporally extended neural decodability windows than lower-level features. Thirty-four right-handed university students (20 females; age = 23.6 ± 2.1) listened to approximately 22 minutes of an audio recording of the Italian tale Il Drago dalle sette teste while EEG was recorded. The narrative transcript was annotated across multiple levels, from acoustics (e.g., envelope, F0) to pragmatic-discourse (e.g., referential distance, speech type), including word form (e.g., length, frequency), lexical-syntactic (e.g., word class), syntactic operation (e.g., dependency distance, opening nodes), syntactic state (e.g., depth, open nodes), conceptual semantic (e.g., concreteness, living/non-living), distributional semantic (e.g., word embeddings) and lexical-pragmatic features (e.g., pragmatic load, metaphoricity). EEG was acquired using a 64-channel system and minimally preprocessed. Continuous data was segmented into epochs locked to word offset (from −400 to 1200 ms). We fitted a back-to-back regression model[2] to estimate how each feature was represented in the EEG signal at each time point. Group-level significance was assessed using cluster-based permutation tests (one-tailed, n = 5000, α = .001). Relative to word offset, word form features were decodable from −0.012 to 0.692 s (t = 3.77, p < .001); lexical–syntactic features from −0.052 to 1.088 s (t = 4.30, p < .001); syntactic operation features from −0.084 to 1.200 s (t = 4.82, p < .001); syntactic state features from 0.056 to 1.200 s (t = 4.90, p < .001); conceptual semantic features from 0.076 to 0.456 s (t = 2.96, p < .001); distributional semantic features from 0.092 to 1.160 s (t = 3.94, p < .001) and lexical-pragmatic features from 0.052 to 0.496 s (t = 3.24, p < .001). By contrast, discourse-pragmatic features were not reliably decodable. See Figure (https://tinyurl.com/B2B-word-offset). This study provides evidence that multivariate predictive modelling can successfully decode a broad set of finely-grained annotated core language dimensions from EEG data, largely replicating the HDC account[1] in another language, from word form to syntactic state level. We further demonstrated the feasibility of decoding conceptual semantics beyond distributional features and lexical-pragmatics. Notably, these clusters emerged earlier than predicted by the HDC framework and partially overlapped with the latency of the lexical-syntactic level. The null result for the discourse-pragmatic level likely reflects an intrinsic limitation of word-level analysis in decoding textual organisation features. Taken together, these findings suggest considerable parallel processing during language comprehension. Gwilliams et al. (2025). PNAS, 122(42), e2422097122; King et al. (2020). NeuroImage, 220, 117028.
Topic Areas: Syntax and Combinatorial Semantics, Meaning: Discourse and Pragmatics