Poster Presentation

Search Abstracts | Symposia | Slide Sessions | Poster Sessions

Distinct yet neighboring neural populations encode past, future, and surrounding speech context in the human temporal lobe

Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai

Marianne de Heer Kloots¹, Atlas Kazemian², William Turner², Josef Parvizi², Laura Gwilliams²; ¹University of Amsterdam, ²Stanford University

Context is critical for both human and artificial speech comprehension systems. Previous studies have found that both preceding and subsequent context drive perception and interpretation of the incoming speech signal. How the brain uses past context in service of prediction has been well documented; however, how the brain uses future context in service of postdictive updating has been less studied, especially in the context of sentence processing, due to experimental and analytical constraints. Here, we leverage advances in artificial speech systems to model the contribution of past and future context on the neural encoding of speech in the human brain. We study contextualized word representations extracted from a self-supervised speech model (Wav2Vec2), and explore how the model’s representations of words are affected when manipulating what context is available beyond the target word; both in terms of context size (number of context words) and in terms of context type (just past, just future, or surrounding context). We first analyse how these context manipulations affect the model’s representation of speech content from a spoken narrative. We then use this model and its context-manipulated representations as a tool to understand the encoding of past, future, and surrounding context in human cortical activity, as recorded with stereo-electroencephalography during naturalistic speech comprehension. First, we find that beyond-word context is crucial for the model’s extraction of abstract (linguistic) word-level features from its acoustic input: information related to word form, meaning and grammatical category is only decodable from model-internal states when past, surrounding or future context was available at input. Second, context-informed speech model embeddings explain unique variance in the temporal lobe, beyond a baseline that includes acoustic features and isolated word embeddings. This effect manifests within a short time window after target word onset (i.e. during the brain’s encoding of the target word, rather than during the context itself). Finally, activity in different electrodes is best explained by different context types (past, future, surrounding context). These electrodes are left-lateralised, and spatially intermixed in the temporal lobe. Overall, we find that spatially neighboring yet distinct neural populations in the temporal lobe encode representations shaped by different contextual sources (past, future, and surrounding input). This provides key insight into the neural circuitry that integrates multiple forms of contextual information, suggesting that both past and future context are integrated locally to derive robust comprehension. Our data are consistent with a recurrent circuit in the temporal lobe leveraging multiplexed representations of different context types. This work highlights the exciting potential of using self-supervised speech model representations to improve our understanding of speech contextualization in the brain.

Topic Areas: Computational Approaches, Speech Perception