Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Eye gaze and cortical tracking of degraded audiovisual speech
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Xiyuan Li1, Holly Robson1, Guangting Mai2, Patti Adank1; 1University College London, 2University of Cambridge
Introduction: Visual cues from a speaker's face enhance speech perception, particularly in challenging listening conditions. Neural synchronisation to rhythmic fluctuations of speech, e.g., amplitude envelope, is associated with successful speech perception. Previous electroencephalogram (EEG) studies have shown enhanced cortical tracking of the speech envelope when congruent visual speech cues are available, especially when the speech is degraded (e.g., [1][2]). Separately, eye-tracking studies have shown that individuals who fixate more on the mouth show behavioural advantages when recognising degraded sentences (e.g., [3][4]). However, how gaze behaviours relate to cortical tracking of degraded speech in more naturalistic contexts remains understudied. Pioneering work by Tan and colleagues [5][6] has begun to link the two measures, testing whether the enhancement in neural tracking stems from gaze behaviours. But they examined only speech in quiet where audiovisual benefits were limited which may explain why no direct gaze-tracking relationship emerged. Our research examines naturalistic audiovisual speech processing under two types of speech degradation: noise-vocoding (Study 1) and multi-talker babble (Study 2, ongoing), exploring (i) whether audiovisual benefits are reflected in cortical tracking of the speech envelope, and (ii) whether individual gaze strategies modulate neural encoding of speech. Methods: In Study 1, 20 native British English adults listened to naturalistic noise-vocoded stories under three modalities (audiovisual[AV], audio-only[A], visual-only[V]) at two degradation levels (4-band, 8-band), while EEG and eye-tracking were recorded simultaneously. Cortical tracking was estimated using forward encoding models of the speech envelope and audiovisual benefit was assessed using the additive criterion [AV > (A+V)]. Eye-tracking analyses defined two facial regions of interest (eyes, mouth), and participants were clustered based on the Eye-Mouth Index, which indicates the relative fixation preference for mouth versus eyes. We then examined the correlations between gaze measures and cortical tracking accuracy. In Study 2 (ongoing), participants listen to short stories in 3-talker babble at +2 and −2 dB SNR in audiovisual and audio-only conditions. Eight Yes/No questions are presented after each story as a behaviour measure of speech comprehension. In the EEG analysis, we plan to model cortical tracking of additional speech features such as the lip aperture. Results: In Study 1, envelope tracking showed no whole-group level audiovisual benefit. Gaze analyses revealed two distinct viewing strategies: mouth-dominant and eye-dominant. Among mouth-dominant viewers, reduced fixation on the eyes relative to the mouth was associated with enhanced envelope tracking in the more degraded condition. For Study 2, we predict greater audiovisual benefits in the more challenging condition, and that more mouth-focused gaze strategy will be associated with stronger cortical tracking of the target speech. Conclusion: Our preliminary findings reveal a novel link between gaze pattern and cortical tracking of the speech envelope, which may reflect a functional role for mouth-focused gaze for mouth dominant viewers. Together with our ongoing work, the project can offer further insight into understanding how visual attention relates to the neural processing of audiovisual speech in naturalistic contexts. REFERENCES [1] Crosse, https://doi.org/10.1523/JNEUROSCI.1829-15.2015 [2] Crosse, https://doi.org/10.1523/JNEUROSCI.1396-16.2016 [3] Trotter, https://doi.org/10.1044/2021_JSLHR-20-00575 [4] Banks, https://doi.org/10.1044/2021_JSLHR-21-00106 [5] Tan, https://doi.org/10.1016/j.neuroimage.2022.119217 [6] Tan, https://doi.org/10.1162/jocn_a_02044
Topic Areas: Speech Perception, Multisensory or Sensorimotor Integration