Search Abstracts | Symposia | Slide Sessions | Poster Sessions
From predictive structure to emotional inference in continuous speech and music tracking in noisy environments
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Li-Chuan Ku1, Christoforos Souganidis1,2, Noemi Bonfiglio1,3, Nicola Molinaro1,4; 1Basque Center on Cognition, Brain and Language, San Sebastián, Spain, 2HiTZ Basque Center for Language Technology - Aholab Signal Processing Laboratory, University of the Basque Country UPV/EHU, Bilbao, Spain, 3University of the Basque Country – UPV/EHU, San Sebastián, Spain, 4Ikerbasque, Basque Foundation for Science, Bilbao, Spain
Musicians have been associated with enhanced auditory encoding of speech, better speech-in-noise perception, and more precise predictive timing. Studies have further suggested that music engages predictive processing similar to language processing (Di Liberto et al., 2020; Kern et al., 2023). However, whether musical expertise similarly modulates predictive precision across speech and music may depend on the hierarchical level of predictive structure in auditory streams. Higher-level predictive structures in speech and music often rely on distinct non-affective feature spaces, such as word- and note-level surprisal. In contrast, affective feature spaces may be more comparable across domains, as both speech and music can be characterized along shared dimensions of valence and arousal. Nevertheless, little is known about the neural tracking of predictive information across speech and music using non-affective vs. affective feature spaces. Here, we examine (1) whether neural tracking encodes domain-specific predictive structure and (2) whether affective encoding generalizes across speech and music in musicians and non-musicians. We hypothesize that music expertise will modulate acoustic tracking across speech and music domains, whereas higher-level predictive structure (word-/note-level surprisal/entropy) will show stronger domain specificity, with musicians exhibiting enhanced encoding particularly in music conditions. Moreover, we expect both acoustic tracking and higher-level predictions to be modulated by attention, particularly in competing-stream conditions. In contrast, affect-related representations are expected to show more similar patterns across speech and music domains, consistent with shared affective processing (Koelsch, 2014). Twenty-five professional musicians (Mage = 43.28, 13 females) from the Higher School of Music of the Basque Country and 22 non-musicians (Mage = 44.04, 12 females) participated in an MEG experiment with six conditions: single speech, single music, speech-speech, music-music, speech-music, and music-speech streams. Stimuli (~6 min each) were loudness-normalized. In two-stream conditions, participants attended to one stream while ignoring the other, followed by recognition judgments of six phrases or music segments from the attended stream. We applied multivariate temporal response functions (mTRF) to estimate subject-specific linear encoding models relating continuous stimulus features to MEG responses in bilateral superior temporal gyrus. To examine predictive processing across domains and listening environments, models included acoustic (spectrograms, onsets) and higher-level lexical/symbolic predictors. Speech predictors included word onsets and contextual word surprisal/entropy derived from GPT-2 (gpt-2-small-spanish), whereas music predictors included note onsets and contextual note surprisal/entropy estimated using the Anticipatory Music Transformer (Thickstun et al., 2023). Preliminary analyses in single-stream conditions confirmed that acoustic and higher-level predictors significantly improved prediction accuracy (all tmax > 3.96, ps < .001). To examine whether affective representations generalize across speech and music, we will construct models combining low-level acoustic features, affect-related acoustic correlates (RMS energy and F0 variability), and higher-level affective dimensions (valence and arousal estimated from a pretrained wav2vec-2.0-based emotion model). We will compare cross-validated prediction accuracy (Pearson’s r) and TRF amplitudes at peak latencies across groups (musicians vs. non-musicians) and attention (attended vs. unattended) using mixed-effects ANOVAs. Together, the current study tests a hierarchical framework in which auditory cortex transforms continuous auditory streams into predictive and affective representations with distinct degrees of domain specificity.
Topic Areas: Computational Approaches, Speech Perception