Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Perceptual Distances Derived from a Deep Neural Network Predict the P2 Elicited by Words in Accented Speech
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
Shang-En Huang1, Yoowon Jung1, Seana Coulson1,2; 1University of California, San Diego, 2Kavli Institute for Brain and Mind
The present study asks whether the processing demands of accented speech are better characterized via acoustic-phonetic features or by holistic perceptual representations derived from deep neural networks (DNNs). Previous attempts to address this question have compared the effectiveness of perceptual distances derived from DNNs and acoustic-phonetic features in predicting human intelligibility ratings, finding that accented speech processing is better captured by perceptual distances. Here we examined auditory P2 latency and amplitude as 40 native English speakers listened to 320 English sentences recorded by talkers from California, Italy, and China. Analysis of the P2 component time-locked to the first word in each sentence was intended to examine accent processing upon initial exposure, in the absence of sentential context. Using a reference talker framework, we computed item-level ERP difference waves by subtracting each item-level ERP to the Californian reference talker from its counterpart as recorded by each of the other talkers. To characterize the speech signals we began by extracting high-dimensional representations of first-word audio segments recorded by the California reference talker using HuBERT-base model embeddings from 4 layers in the DNN (0, 4, 8, and 12). Analogous embeddings were extracted for the other talkers and the perceptual distances of each talker/item from that of the reference talker were computed using cosine distance after dynamic time warping. These perceptual distance measures were used to predict the P2 effects and compare models of the same data using the acoustic-phonetic measures of Mean F0 and MFCC distance. Mixed-effects modeling revealed that, for sentence-initial words, regression models incorporating HuBERT-derived distances provided a better fit than those based on word-level acoustic features. Perceptual distances extracted from lower layers emerged as significant predictors of auditory P2 responses after FDR correction, with greater perceptual distance associated with smaller P2 amplitudes (p < .05) and delayed latencies (p < .001); by contrast, acoustic predictors showed weaker or non-significant effects. Similar analyses of the P2 elicited by the last word in each sentence revealed no significant effects either for perceptual distance using word-level HuBERT embeddings or for acoustic predictors. Next we asked whether the P2 elicited by sentence final words was more sensitive to the cumulative impact of unfamiliar speech sounds in the context preceding the last word. Accordingly, we tested whether distance measures using full-sentence HuBERT embeddings provided a better basis for predicting P2 to sentence-final words than did isolated representations of the words. Indeed, perceptual distances derived from the full sentence reliably predicted last-word P2 amplitude (p < .05), with greater distances associated with larger P2 amplitudes. Finally, P2 amplitudes were modeled with sentence-level acoustic-phonetic features including Mean F0 (extracted from syllable nuclei), Number of Pauses, and Articulation Rate. Results showed that perceptual distance better accounted for P2 amplitude than the acoustic model, especially using embeddings from the highest layer (ΔAIC = 9.96). These findings suggest sentential context has a marked effect on the cortical auditory response to speech and support speech perception models that incorporate holistic representations.
Topic Areas: Speech Perception,