Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Extending speech tracking to fMRI: Neural responses to sentence boundaries in spontaneous speech
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
Galit Agmon1, Orel Levy1, Refael Tikochinski2,3, Roi Reichart4, Isabelle Deschamps5; 1Bar-Ilan University, Israel, 2University College London, 3Camebridge University, 4Technion – Israel Institute of Technology, 5Georgian College, Ontario, Canada
**Introduction: Sentence boundaries are a fundamental organizing feature of language, marking points at which one syntactic unit is completed and another begins. Yet direct neuroimaging evidence for the identification of such boundaries remains limited. A major challenge is that syntactic boundaries are difficult to isolate experimentally without relying on artificial stimuli. Here, we address this challenge using naturalistic spontaneous speech. In a whole-brain fMRI approach inspired by the speech-tracking literature in electrophysiology, we simultaneously modeled speech comprehension with multiple continuous regressors, thereby isolating sentence-boundary processing from other variables. **Methods: We analyzed fMRI data from 36 participants from the publicly available Narratives dataset (Nataste et al. 2021), who listened to spontaneous speech during scanning (TR = 1.5 s, voxel-size = 3×3×4 mm). Sentence boundaries were manually annotated based on linguistic criteria and served as the main regressor of interest in a general linear model (GLM). The GLM was designed to distinguish boundary-related responses from acoustic, prosodic, and linguistic features. Word onset was included as a binary regressor. A low-level acoustic regressor was extracted from the speech envelope, convolved with a canonical HRF (4.5 s peak lag, estimated through cross-correlation with primary auditory cortex activity), and downsampled to the fMRI sampling rate. Because sentence endings are often accompanied by prosodic cues, we included regressors for prosodic break and emphasis, calculated using a wavelet-based toolkit. Node count, computed through bottom-up tree traversal, was included to test whether boundary-related responses extend beyond incremental syntactic structure building. Additional linguistic regressors included word frequency, surprisal, and concreteness. Collinearity was low across the design (all VIFs < 2). Whole-brain voxelwise GLMs were conducted in AFNI, followed by group-level analyses controlling for participants’ age, behavioral performance, and experimental condition. **Results: Reliable tracking of the acoustic envelope was observed already at the single-subject level, indicating that rapidly fluctuating acoustic features can be robustly captured with fMRI. At the group level, the acoustic envelope revealed bilateral superior temporal activity, consistent with canonical auditory speech-processing regions. The concreteness regressor elicited bilateral responses in the anterior temporal lobe, consistent with previous literature. Prosodic features activated the superior temporal gyrus bilaterlally. Both word onset and surprisal elicited widespread responses across language-related regions, predominantly in the left hemisphere. Together, these results validate the sensitivity of our approach, showing that whole-brain fMRI can capture fine-grained temporal fluctuations in both acoustic and linguistic properties during natural speech comprehension. Within this design, sentence boundaries elicited a distinct positive response along the right MTG. This effect was observed after accounting for prosody and bottom-up syntactic structure building, suggesting an additional integrative “wrap-up” process at sentence-final boundaries. **Conclusions: Our results suggest that sentence boundaries are tracked in fMRI beyond their prosodic realization or bottom-up syntactic node count. The right-hemisphere localization of this effect is consistent with proposals that the right hemisphere is preferentially sensitive to information unfolding over longer temporal timescales. More broadly, we demonstrated that speech-tracking approaches can be extended to whole-brain fMRI, opening a route for studying natural language comprehension as it unfolds in ecologically valid speech.
Topic Areas: Speech Perception, Syntax and Combinatorial Semantics