Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Hierarchical Speech Representations in Whisper Align with MEG Responses During Natural Speech Listening
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Itsuki Hamada1, Yushi Sugimoto2, Masahiro Yamashita3, Shinji Nishimoto2,3, Hiromu Sakai1; 1Waseda University, 2The University of Osaka, 3Center for Information and Neural Networks (CiNet)
Speech comprehension requires hierarchical processing, converting acoustic signals into linguistic representations. Large-scale speech models such as Whisper (OpenAI) have recently attracted attention as computational models of this hierarchy, as their encoder layers progressively transform acoustic information into increasingly abstract representations (Radford et al., 2023). The hierarchical representation should be universal across language. However, evidence for such alignment has mainly come from English fMRI and ECoG studies, leaving MEG studies and non-English languages underexplored, despite substantial cross-linguistic differences in speech features. Here, we examined whether layer-wise Whisper encoder representations predict MEG responses during natural Japanese speech listening , and which layers best capture cortical speech processing. Two native Japanese speakers listened to 23 approximately 10-minute stories from the Corpus of Spontaneous Japanese (Maekawa et al., 2014) while whole-head MEG was recorded using an Elekta Neuromag® 360 system. Analyses were performed on 240 gradiometers. MEG data were preprocessed with tSSS, 0.1–100 Hz band-pass filtering, and downsampling to 200 Hz. We compared three feature sets: envelope (80–8000 Hz), log-mel spectrogram, and hidden states from all 13 layers of the Whisper-small encoder, with each layer reduced to 50 dimensions using PCA. For each feature set, we estimated temporal response function (TRF)-based encoding models with time lags from −500 to 1000 ms using ridge regression on 21 training stories. The regularization parameter was selected by leave-one-story-out cross-validation. Model performance was evaluated on two held-out test stories using Pearson’s correlation coefficient r. Each test story was presented twice, and the two MEG responses were averaged. Significance was assessed by a permutation test using random temporal shifts of at least 10 s between measured and predicted responses, with FWER correction across channels. Both Whisper-based and low-level acoustic models significantly predicted MEG responses. After FWER correction, significant prediction accuracy was found in at least 117/240 channels for both participants and all feature sets, reaching 200/240 channels for Whisper layer 7 in participant 2. In both participants, many Whisper layers outperformed the envelope and log-mel spectrogram in mean prediction accuracy across gradiometers; this advantage was observed in all layers except layer 0. Prediction accuracy increased from shallow to middle layers, but did not increase monotonically across the entire encoder depth. This layer-dependent prediction profile and the advantage over low-level acoustic features indicate that Whisper-derived representations reflect neural responses beyond acoustic processing alone. The highest mean prediction accuracy was observed in layer 12 for participant 1 (mean r = 0.067, max channel r = 0.223) and in layer 7 for participant 2 (mean r = 0.095, max channel r = 0.343), with the maximum-prediction channels located over temporal regions in both participants. These results suggest that Whisper encoder representations capture the hierarchical nature of cortical processing of Japanese speech, suggesting that such hierarchical speech representations can be observed across languages. Future work will compare these results with higher-level linguistic features, including LLM representations, and examine TRF time-response components and source-level estimates.
Topic Areas: Computational Approaches,