Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Time-varying modulation representations as a model of cortical speech processing measured by magnetoencephalography
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Chang Yan1,2, Anni Nora1,2, Marijn van Vliet1,2, Riitta Salmelin1,2; 1Aalto University, 2Aalto NeuroImaging (ANI)
Speech is a unique human ability, enabling efficient communication between individuals. Studying the complicated correspondence between acoustic signals and their neural representations would provide valuable guidance for applications in technology and clinical practice. A recent magnetoencephalography (MEG) study found that the brain tracks speech more closely in time compared to non-speech sounds (Nora, et al., Eneuro, 2020, 7(4)). Whereas that study looked at linear tracking of the amplitude modulations, analysis of intracranial data has indicated that the varied temporal fluctuations in speech require both linear and nonlinear representations in their cortical processing (Pasley, et al., PLoS biology, 2012, 10(1)). Here, we investigate whether using non-linear features improves the decoding of speech sounds from noninvasive MEG recordings, especially regarding the fast spectrotemporal modulations within speech, reflecting e.g., phoneme onsets. We analysed data from 16 Finnish-speaking participants who listened to 44 spoken words and non-speech sounds with corresponding meanings (Nora, et al., Eneuro, 2020, 7(4)). Assuming that neural responses can predict past sounds, we employed lag windows ranging from 20 ms to 420 ms to capture the latencies when sound features are represented in brain activity. The linear features included averaged amplitude envelope and amplitude modulations at each frequency band of the full spectrogram, whereas the nonlinear features were time-averaged and time-varying modulation power spectrum (MPS). For the MPS, combinations of predefined temporal rates (1, 3, 9, 27 Hz) and spectral rates (0.5, 1, 2, 4 cyc/oct) were chosen. A convolution model was used for decoding these time-varying features and its performance was evaluated using leave-two-out cross-validation. We found that for high temporal rates (~27 Hz), the decoding accuracy using MPS feature was higher than when using linear features. This indicates that for encoding fast acoustic-phonetic features such as syllable onset and offset, a nonlinear cortical representation is needed. In contrast, linear spectrogram and amplitude envelope representations had higher decoding accuracy at low temporal rates(~3Hz), which implies that important slow structural features like syllable rhythm and formants of natural speech are encoded as cortical evoked activation that linearly follows the rising and falling of the amplitude changes. For all three sets of speech features, the best accuracy mostly falls at 20-180 ms lag windows, showing that the evoked activation tightly follows the speech sounds in time. This similar pattern of decoding results was seen based on responses in both left and right temporal regions. Non-speech sounds were better decoded based on responses in the right than left temporal areas, with statistically significant hemispheric difference seen mostly for slow temporal rates (~3Hz), but their decoding did not exhibit time-locking comparable to that of speech. Based on these results, we conclude that non-linear features of the spectrotemporal modulations during speech are indeed reflected in the evoked responses measured by noninvasive MEG recordings.
Topic Areas: Methods, Speech Perception