Search Abstracts | Symposia | Slide Sessions | Poster Sessions
How visual speech salience and linguistic informativeness shape audiovisual lexical-semantic processing
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
This poster is part of the Sandbox Series.
David Hernández-Gutiérrez1, Aaron Nidiffer1, Jin Dou1, Edmund Lalor1; 1University of Rochester, NY, USA
In face-to-face language communication, the visible cues provided by a speaker’s orofacial movements -visual speech- can facilitate both acoustic and linguistic processing. Neuroscientific evidence has revealed audiovisual facilitation during early stages of speech processing, including shorter latencies of sensory event-related potentials (ERPs) to phonemes whose visual counterparts -visemes- are highly recognizable (van Wassenhove et al., 2005). The case of lexical-semantic processing, however, has yielded mixed findings. While some studies have not found differences in N400 amplitude or latency as a function of sensory modality (Hernández-Gutiérrez et al., 2018), others have reported audiovisual effects (e.g., Brunellière et al., 2013; Gastaldon et al., 2026). One possibility is that these linguistic effects also depend on the visual saliency of visemes (Brunellière et al., 2013). However, although sufficient to facilitate speech encoding, visual saliency alone is unlikely to fully account for higher-order linguistic processing. A second requirement may be the capacity of visemes to reduce uncertainty during lexical selection, as the visual advantage provided by visemes may depend on the complementary information they provide to constrain lexical competition and facilitate lexical access (Nidiffer & Lalor, 2024). Because viseme-to-phoneme mappings are not one-to-one, visual constraints may evolve over the course of a word. Although previous work has shown evidence of visual lexical competition, its impact on subsequent lexical-semantic encoding remains unclear. The present work will test a constraint-based model of audiovisual language processing in which visual speech confers a lexical-semantic advantage only when two requirements are met. First, articulatory positions must be sufficiently visible to support viseme recognition. Second, visemes must reduce uncertainty over lexical candidates given the valid phonemic continuations, thereby constraining lexical access and modulating downstream lexical-semantic processing. To address this question, we will reanalyze the publicly available AVbook EEG dataset (Varano et al., 2023, 2026), in which neural activity was recorded from 19 participants while they listened to an audiobook in background noise. Specifically, we will focus on three sensory conditions: audiovisual (AV), auditory-only (A), and visual-only (V). In the V and AV conditions, participants were presented with a video of the speaker’s face. Temporal response functions (TRFs) will be used to characterize EEG responses to both acoustic and linguistic features, including the speech envelope, word onset, cohort entropy, and surprisal across sensory conditions. A multisensory integration index will further quantify the extent to which AV reconstruction accuracy exceeds that predicted by an A+V model (Crosse et al., 2015). Finally, to evaluate the impact of visual saliency and lexical competition on the temporal facilitation of lexical-semantic encoding, we will apply the dynamic-TRF framework proposed by Dou et al. (2025). We predict that N400-like TRFs will show shorter latencies and reduced amplitudes in the AV condition, but selectively for words with articulations that are easily perceived and when the corresponding viseme reduces lexical competition. Such findings would provide mechanistic support for a constraint-based account of audiovisual lexical-semantic processing and shed light on how the brain differentially exploits visual cues across distinct levels of linguistic processing.
Topic Areas: Multisensory or Sensorimotor Integration,