Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Phoneme overlap analysis of the visual benefit for speech-in-noise perception
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
John Magnotti1, Yue Zhang2, Lin Zhu3, Yingjia Yu4, Michael Beauchamp1; 1University of Pennsylvania, 2Baylor College of Medicine, 3University of Texas Southwestern Medical Center, 4University of Pittsburgh
Seeing the face of the talker improves the intelligibility of speech-in-noise, termed the visual benefit. The discovery that both voice and face contribute to speech perception has spurred thousands of studies in linguistics, psychology, neuroscience, and computer science. The visual benefit is usually measured at the whole-word level, with responses classified as "correct" or "incorrect". An alternative method measures visual benefit using the phoneme overlap between stimulus and response. In 55 participants presented with auditory and audiovisual words, phoneme-overlap analysis estimated 289 different visual benefit values, compared with only three different values for whole-word analysis. Phoneme-overlap detected a positive visual benefit in 67% of word-participant pairs, in contrast to 32% for whole-word analysis. Individual words varied greatly in their phoneme-overlap visual benefit, from 1% to 72%. Variability across words was accurately predicted (r = 0.78) by the phoneme composition of the word and the visual benefit for untested words could be predicted with a median error of 9%. Measuring the visual benefit for individual phonemes revealed similarly high variability (1% to 70%) with six phonemes (W, F, V, TH, P, AW) exhibiting visual benefit more than one standard deviation above the mean. The increased sensitivity of phoneme-overlap analysis relative to conventional whole-word analysis may provide a useful tool for investigating audiovisual speech perception in clinical, neuroscience and computer science contexts. For instance, intracranial recordings from human patients have demonstrated the existence of small populations of neurons in temporal lobe that are selective for different groups of phonemes during auditory presentation of speech. In a word-level analysis, observing the face of the talker increased response and decreases response latency in the superior temporal sulcus. If visual benefit varies across phonemes, a phoneme-level analysis might reveal differential modulation of neuronal responses depending on the phoneme content of individual stimulus words. Incorporating expected differences in visual information may also improve the fit of EEG-based methods that rely on a reconstruction of the measured brain signal, capturing variance that otherwise would be assigned to unmodelled error.
Topic Areas: Speech Perception, Phonology