Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Slide Session B: Fine-Scale Dynamics of Speech Perception
Wednesday, September 30, 3:30 - 4:30 pm, Amphitheatre D
Talk 1: Lexical representations are sequenced through a micro-scale dynamic code in inferior frontal gyrus
Irmak Ergin1, Atlas Kazemian1, Ernesto Rojas1, Nick V Hahn1, Foram Kamdar1, Christina Kim Vo1, Sasi S Madugula1, Ryan Z Wang1, Chaofei Fan1, Erin M Kunz1, Donald T Avasino1, Erxiao Wang1, Yuhan Zhang1, Cory Shain1, Leigh R Hochberg2,3, Jaimie M Henderson1, Francis R Willett1, Laura Gwilliams1; 1Stanford University, 2Brown University, 3Massachusetts General Hospital
Naturalistic speech is transient and strictly time-ordered, creating a computational challenge for comprehension: the brain must preserve and integrate earlier input while new words continue to arrive. One proposed algorithmic solution is that linguistic representations are maintained dynamically across changing neural populations over time (Gwilliams et al., 2025; Zhang et al., 2026). This implements a long-lived dynamic neural code, which enables parallel processing of successive inputs without inference between neighbours, and implicitly keeps track of their order in the sequence. While this phenomenon has been documented in large neural populations for lower-level representations, how it is implemented at the neuronal scale for higher-level linguistic representations remains unclear. To address this, we analyzed micro-scale neural activity from Utah arrays (3.5 × 3.5 mm grids) placed in inferior frontal gyrus (IFG) of two BrainGate2 clinical trial participants with ALS while they listened to stories (Huth et al., 2016). First, we tested whether the micro-scale patterns of activity in IFG encode higher-level speech information. We used logistic regression classifiers to decode Part of Speech (PoS; nouns, adjectives, and verbs) and found significant decoding of all PoS categories in both subjects. Notably, decoding performance persisted above chance for over 2s after word onset, far exceeding the average word duration. Second, we tested whether the encoding of PoS was supported by a dynamic code at this scale. Temporal generalization analysis revealed that neural patterns at one time-step generalize poorly to later time steps, indicating that the underlying neural activity patterns that encode PoS evolve over time. Third, we examined the underlying neural implementation of this dynamic code. Peristimulus time histograms (PSTHs) locked to word and sentence onsets revealed a peak response latency range of approximately −0.5 to 1.5s. This suggests that nearby neural populations respond to the same information at different times, supporting the idea that linguistic representations are carried forward by successive populations as processing evolves. Finally, we summarized the PSTH responses at the population level using PCA. We found that just two components explained over 80% of the variance in the average responses to words and sentences. When projecting the PSTHs onto these two axes, responses traced a smooth state-space trajectory. Our findings provide the first characterization of the micro-scale dynamics in human IFG that sequence lexical representations during natural listening. This dynamic code is implemented with three key factors: representational redundancy, temporal heterogeneity, and longevity. Nearby neural populations - recorded simultaneously and from electrodes that are 400 microns apart - respond to word onsets, and encode similar lexical information. The responses at different electrodes, however, vary widely in latency - some peak within tens of milliseconds, and others several hundred milliseconds. The consequence is a dynamic neural pattern that encodes both linguistic content and relative time. Together, this suggests that a word's location along the neural population trajectory may act as an implicit time code for its place in the local sequence, allowing ordered assembly of higher-order structure.
Talk 2: Neural hierarchy for auditory and speech perception
Magdalena Kachlicka1,2, Alexandros Christopoulos1,2, David Ricardo Quiroga Martinez3,4, Jon T. Willie5,6, Peter Brunner5, Robert T. Knight7,8, Athina Tzovara1,2; 1Institute of Computer Science, University of Bern, Switzerland, 2Center for Experimental Neurology and Sleep Wake Epilepsy Center-NeuroTec, Department of Neurology, Inselspital, Bern University Hospital, University of Bern, Switzerland, 3Center for Music in the Brain, Department of Clinical Medicine, Aarhus University & The Royal Academy of Music Aarhus/Aalborg, Aarhus, Denmark, 4Department of Psychology, University of Copenhagen, Copenhagen, Denmark, 5Department of Neurosurgery, Washington University School of Medicine in St. Louis, USA, 6Department of Neurosurgery, Dell Medical School, University of Texas at Austin, USA, 7Department of Neuroscience, University of California, Berkeley, USA, 8Department of Psychology, University of California, Berkeley, USA
Speech perception requires integration of acoustic and semantic information across multiple scales, from individual phonemes and syllables to words and full sentences. Models of speech processing describe a cortical organisation in which primary and secondary auditory areas in the lateral temporal lobe perform spectrotemporal analysis, and word recognition (Chang et al., 2010; Zhang et al., 2026). Further processing propagates along frontal, parietal, and temporal association areas to support lexical access, syntactic parsing, and semantic integration (Hickok & Poeppel, 2007; Ding et al., 2016). There is substantial evidence for involvement of the insular complex in speech perception, including functional separation between anterior and posterior insula (Blenkmann et al., 2019; Kurteff et al., 2024; Nourski et al., 2022), suggesting distinct roles within this hierarchy. However, a direct comparison of neural responses across the extended auditory network and across different levels of linguistic context is lacking. In this study, we characterise intracranial electroencephalography (iEEG) responses to speech (isolated syllables, words starting with those syllables, and sentences containing those words) and non-speech (pure and complex tones) stimuli, allowing direct comparison of responses to the same material embedded in progressively richer contexts. We recorded iEEG activity from 20 patients undergoing presurgical monitoring for drug-resistant epilepsy, with electrode coverage concentrated in the lateral temporal lobe and additional electrodes in parietal and frontal cortices, medial temporal lobe, and the insula. We extracted intracranial event-related potentials (iERPs; 1-40 Hz) and high-frequency activity (HFA; 80-150 Hz) and compared regional response onsets and peak latencies using permutation tests. We also examined clusters of prototypical responses to stimuli embedded in different lexical contexts by investigating spectrotemporal patterns across frequency bands. We found that tones elicit earliest iERP onsets from subcortical structures (Me=57 ms), through lateral temporal lobe (61.25 ms), and insula (63 ms), to latest responses in parietal (81.5 ms), frontal (98.5 ms), and medial temporal lobes (112.75 ms). A processing hierarchy for speech parallels that for non-speech stimuli with the fastest onsets in subcortical structures (99.57 ms), insula (175.13 ms) and lateral temporal lobe (175.78 ms) followed by parietal (296.81 ms), frontal (315.69 ms), and medial temporal areas (488.75 ms). Responses to speech in medial temporal lobe showed progressively later onsets and peaks with increasing linguistic complexity. We also observed that most of the electrodes in lateral temporal lobe and insula that were responsive to non-speech stimuli maintained their activity to speech stimuli across conditions. This was not the case for electrodes in frontal and parietal lobes and subcortical structures, which were predominantly responsive to non-speech stimuli. Clustering analysis revealed patterns of transient onset responses in the theta band (4-8 Hz) and sustained gamma (30-80 Hz) and high-frequency (80-150 Hz) activity across conditions. Our findings provide a more complete characterisation of the neural hierarchy underlying human auditory perception, extending from auditory cortex to insula, parietal, frontal, and medial temporal areas. We further confirm the active role of the insula in processing both speech and non-speech stimuli.
Talk 3: Non-hierarchical predictive processing of acoustic and semantic content in speech
Chang D. Liu1,2,3, Silke Ethofer4, Jonas Hebel5,6, Georgios Naros4, Martin Holtkamp5,6, Randolph F. Helfrich7,8, Yulia Oganian1,2,3; 1Centre for Integrative Neuroscience, University Medical Center Tübingen, 2Graduate Training Centre of Neuroscience, University of Tübingen, 3International Max Planck Research School “Mechanisms of Mental Function and Dysfunction”, Tübingen, 4Department of Neurosurgery, University Medical Center Tübingen, 5Charité-Universitätsmedizin Berlin, Department of Neurology Berlin, 6Epilepsy-Center Berlin-Brandenburg, Berlin, 7Department of Psychology, Yale University, 8Wu Tsai Institute, Yale University
Predictive processing is key to human speech comprehension. To date, models of predictive language processing have followed the predictive coding framework. In this view, comprehension is guided by differences between expected and perceived inputs, with hierarchical propagation of such prediction errors across representational levels. Alternatively, predictive speech processing might rely on the joint information content across representational levels, without assuming a predictive hierarchy. Notably, the rich spectrotemporal dynamics of speech, including cues at sub-phonemic temporal scales, are predictive of semantic content. They are thus a perfect testcase of how predictive information is combined across acoustic and semantic levels. We capitalized on a recently developed transformer-based autoregressive text-to-speech model (Fish-Speech 1.5) to obtain co-occurrence based surprisal of acoustic and semantic tokens, as well as estimates of the conditional surprisal of acoustic tokens given semantic content (henceforth speech surprisal). From these measures we derived the pointwise mutual information (PMI). PMI is a symmetric measure of sound-meaning association strength, which does not assume a predictive hierarchy, defined as the difference between acoustic and speech surprisal. A corpus analysis of spontaneous conversations showed that PMI was higher for lexico-semantic tokens than for non-syntactic pragmatic markers, such as hesitations. Moreover, PMI was also higher at word and phone onsets than in word/phone-medial positions, respectively. This analysis revealed the potential informativeness of PMI for speech comprehension. We next asked whether PMI is represented in the cortical speech processing network. If so, neural activity should increase with acoustic surprisal and decrease with speech surprisal, i.e. respond to acoustic cues that are rare but expected given semantic information. In contrast, under hierarchical predictive coding, neural activity should be maximal when acoustic cues are common but unexpected given semantic information. That is, neural activity should decrease with acoustic surprisal but increase with speech surprisal, predicting the opposite neural response pattern. We recorded cortical activity using intracranial EEG (ECoG and sEEG), while participants listened to recordings of spontaneous conversations (n = 12 patients, 75 speech-responsive electrodes). We complemented our data with a publicly available ECoG dataset, where participants listened to a podcast (n = 9 patients, 294 speech-responsive electrodes. Encoding models revealed encoding of acoustic, speech, and word surprisal in the superior temporal gyrus, whereas neural responses in the inferior frontal gyrus encoded only word surprisal. Encoding of acoustic and speech surprisal was highly correlated, indicating that they were jointly encoded by the same neural populations. Notably, on over 90% of this population, neural responses increased with speech surprisal but decreased with acoustic surprisal. That is, response profiles reflected PMI rather than a hierarchical prediction error. Furthermore, as predicted by the corpus analysis, PMI-encoding populations responded stronger to words than to hesitations. Similarly, neural responses to word and phone onsets were modulated by PMI. Overall, our results suggest that cortical speech processing relies on information filtering via joint analysis of acoustics and lexico-semantic content, in addition to predictions of acoustic and lexico-semantic content.
Talk 4: Single-neuron and population-level encoding of content and time in the human speech cortex
Aakash Sarkar1, Shailee Jain1, Yitzhak Norman1, Stephanie Hu1, Rujul Gandhi2, Paulo Parramon-Arcos1, Matthew K. Leonard1, Edward F. Chang1; 1University of California San Francisco, 2Harvard University
Introduction. Comprehension of natural speech requires the auditory system to map rapid acoustic signals onto linguistic representations while simultaneously tracking sequence structure in time - implicating a code that integrates content ("what") with sequence timing ("when"). While macro-scale electrophysiology has established the superior temporal gyrus (STG) as critical for phonetic feature extraction, single-unit mechanisms tracking temporal sequences during continuous speech remain unclear. Methods. We recorded single-unit spiking (N > 1,500 STG neurons) via high-density Neuropixels from 17 awake human participants passively listening to sentence-level TIMIT (50 ms sentence-onset-aligned PSTHs). We fit ridge-regression Temporal Receptive Field (TRF) models combining acoustic, phonetic, and HuBERT-derived temporal-position features. We applied time-similarity analysis, PCA, and Isomap to population activity to characterize representational geometry, and trained ridge decoders for temporal reconstruction. Results. STG neurons exhibit widespread mixed selectivity; comparing TRF models across feature classes identified a population of ‘multiplexer’ cells jointly tuned to phonetic categories at distinct temporal positions, paralleling individual neurons responding to approximants (e.g., /l/, /r/) solely at speech onset. These combined models explained ~40% of variance in held-out test sets, significantly outperforming isolated feature models. A subpopulation of 'time cells' exhibits compressed sequential firing (J-shaped in linear time, straightening logarithmically), with time-field widths scaling linearly with peak latency and a power-law decay in peak density - qualitatively reproducing the temporal-coding pattern of hippocampal CA1 time cells (Cao et al., 2022). This sequential structure is preserved across distinct sentences (content-invariance). A distinct class of ‘lexical boundary cells’ (n ≈ 50) explicitly encodes boundaries while remaining acoustic-agnostic, showing characteristic dips at word transitions. This provides a candidate single-unit substrate for population-level cortical "resets" observed in macroscopic STG recordings (Zhang et al., 2025). Time-similarity analysis reveals temporal recurrence blocks mapping onto phoneme and word durations. Furthermore, PCA reveals sentence-level macro-loops across the population, word-level loops in boundary cells, and combined timescales in multiplexer cells (word-level sub-loops embedded within larger sentence-level macro-loops). Population-level Isomap projections of multiplexer cells reveal a low-dimensional manifold sheet disentangling time from content - where elapsed time forms a smooth longitudinal backbone and phonetic variance drives orthogonal expansion, and corresponding Isomap projections of boundary cells introduce sharp inflection points at these lexical transitions. Ridge decoders trained on STG activity reconstruct elapsed time within sentence near-perfectly (example R ≈ 0.999) and generalize across a 60×60 novel sentence-pair matrix (strongest pairs reaching R > 0.4, R² > 0.15), indicating STG maintains a generalizable temporal coordinate system independent of phonetic content. Conclusion. These results provide single-unit evidence that the human STG integrates phonetic content with temporal sequence structure during natural speech. This includes compressed time-cell sequences sharing the linear width–peak scaling and power-law peak density of hippocampal CA1 time cells, alongside acoustic-agnostic word-level dynamics consistent with macroscopic boundary resets. Word- and sentence-level structures emerge at distinct population timescales, extending hippocampal frameworks to human sensory cortex and demonstrating geometric disentanglement of content and time during naturalistic language.