Search Abstracts | Symposia | Slide Sessions | Poster Sessions
How do musicians hear and see the rhythm of speech? A MEG investigation on cortical tracking of natural speech in noise in the auditory and visual modality
Poster Session A, Wednesday, September 30, 11:00 am - 1:00 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Noemi Bonfiglio1,2, Romain Pastureau1,2, Nicola Molinaro1,3; 1Basque Center on Cognition, Brain and Language, Donostia-San Sebastián, Spain, 2University of the Basque Country – EHU, Donostia-San Sebastián, Spain, 3Ikerbasque, Basque Foundation for Science, Bilbao, Spain
Extracting rhythm from continuous stimuli is a crucial skill for both successful music performance and effective speech processing. In line with this, it has been proposed that more efficient rhythm processing underlies the observed advantages in speech-in-noise perception in expert musicians. Previous studies found that cortical tracking of speech is indeed stronger in musicians, especially in difficult listening conditions, which indicates higher neural sensitivity to timing information in the speech stream. However, most research on music expertise and speech processing focused on auditory-only perception, while everyday face-to-face communication is heavily multimodal: we continuously integrate auditory and visual input to infer the intended meaning from the speaker, and such integration is particularly beneficial in noisy situations. Crucially, during face-to-face communication, rhythm processing is not limited to the auditory modality, as both head and upper-limb movements behave as additional rhythmic oscillators temporally aligned to the speech. Therefore, increased timing sensitivity given by musical practice might affect 1) the processing of rhythmic visual cues such as hand movements (hereafter, “gestures”) and 2) the integration between auditory and visual information when they are temporally congruent. In this study, we explore these two processes in musicians (n=24) and control non musicians (n=23) through measures of cortical tracking in the auditory and visual domain. Data collection has been completed, while the initial stage of data analysis is currently being carried out. In a MEG experiment, participants attended videos of individuals spontaneously retelling cartoons in three conditions: Audio-Visual (AV), Audio-only (A) and Visual-only (V). In both AV and A conditions the speech stream was embedded in naturalistic noise (cocktail party crowd). Speakers’ gestures were recorded through a Kinect system, and the extracted movement parameters will be used to quantify tracking in the visual domain. Auditory and visual tracking will be measured with Temporal Response Function (TRF) encoding models, which allow to isolate the unique contributions of auditory and visual regressors to the prediction accuracy (r) of the model. To ensure that TRF models are comparable across conditions, we will use a subtractive approach. First, a full model will include both auditory (i.e., spectrogram) and visual (i.e., gestures’ speed) regressors. Then, we will quantify auditory and visual tracking as the difference in prediction accuracy (Δr) between the full model and two reduced models with scrambled auditory and visual regressors, respectively, in each experimental condition. Finally, we plan to run ANOVAs on auditory and visual Δr, with group and condition as independent factors. We expect that 1) if musicians display increased timing sensitivity across modalities, we will observe an effect of group in both auditory and visual tracking; 2) if audiovisual integration is different in musicians, we will observe interactions between group and condition (i.e., the difference in tracking between AV and A/V will differ between groups). Overall, this investigation will shed new light on the effects of music expertise on the neural processing of speech, testing cortical tracking across modalities and in more ecological conditions than previously considered.
Topic Areas: Multisensory or Sensorimotor Integration, Signed Language and Gesture