Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Generalization of Linguistic Representations Across Low- and High-Variability Speech Perception Regimes
Poster Session F, Friday, October 2, 2:45 - 4:45 pm, Wangari Maathai
Francesco Mantegna1, Dulhan Jayalath1, Gereon Elvers1, Tasha Kim1, Benjamin Ballyk1, SungJun Cho1,2, Teyun Kwon1, Luisa Kurth1, Miran Özdogan1, Gilad Landau1, Pratik Somaiya1, Natalie Voets1,3, Mark Woolrich1,2, Oiwi Parker Jones1; 1University of Oxford, 2OHBA, Oxford Centre for Human Brain Activity, 3FMRIB, Oxford Centre for Functional MRI of the Brain
Communication relies on shared reference: we understand one another because we can converge on the sound of a phoneme or the meaning of a word. The ability to converge on shared linguistic references suggests some alignment between linguistic representations. Yet it remains unclear how stable these representations are across contexts within the same individual and across individuals. Characterizing stable linguistic representations is challenging because variability arises at multiple levels. The spectrotemporal features of the same word can vary across speakers, accents, and linguistic contexts. Moreover, the same linguistic content may be represented differently across individuals as a function of trait variables, such as semantic knowledge, and state variables, such as mood. Anatomical and functional differences across individuals may further interact with these linguistic factors and increase variability. Previous work suggests that invariant linguistic representations can be recovered from neurophysiological data and, to some extent, generalized within and across individuals. Naturalistic speech perception paradigms are well suited to recovering stable linguistic representations because they expose participants to many stimuli, enabling large-scale analyses. In this context, the performance of linear methods tends to plateau, whereas deep learning methods can benefit from scaling: with more data and hyperparameter tuning, performance can continue to improve. Such scaling suggests that models can exploit shared structure across repeated instances of phonemes and words. However, it is unclear how scaling is affected by variability within and across participants. In a series of magnetoencephalography studies, we investigated scaling during naturalistic speech perception. In the first study, we adopted a controlled, low variability setting: one participant listened to the entire Sherlock Holmes canon narrated by a single speaker, yielding approximately 50 hours of recordings. A forced alignment algorithm was used to generate word- and phoneme-level annotations from the audio and transcript. In this setting, even simple deep learning models showed scaling behavior across multiple tasks, including phoneme classification and word classification. In a follow-up study, we moved to a heterogeneous, high variability setting by expanding the dataset to include different linguistic materials and participants, for approximately 100 hours of recordings. The materials combined corpora with distinct properties: Sherlock Holmes audiobooks, which are relatively semantically homogeneous and narrated by a single speaker; TIMIT, which contains isolated sentences spoken by many speakers with diverse voices and accents; and podcasts, which contain spontaneous autobiographical narratives spanning diverse semantic topics and speakers. The dataset also included thirty-two participants differing in age, sex, and native language. Preliminary results in this high variability regime suggest that generalization of linguistic representations across speakers, materials, and participants is possible, but not trivial. Achieving scaling under increased variability appears to require more careful choices of model architecture and hyperparameter tuning, together with modeling approaches that explicitly account for distribution shifts across speakers, materials, and participants. Overall, these findings suggest that linguistic representations exhibit some alignment within and across individuals, while recovering these shared representations under increasing variability requires more sophisticated methods to preserve scaling and support generalization. The dataset and accompanying data loaders are publicly released.
Topic Areas: Speech Perception, Methods