Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Audiovisual sentence processing in adult readers: Neural correlates and relation to phonological awareness
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Rahel Wombacher1,2, Alexandra Schreiber1,2, Hanna Rutschmann1,2, Iliana I. Karipidis1,2, Amelie Haugg1,2, Silvia Brem1,2; 1University of Zurich, 2University Hospital of Psychiatry, University of Zurich, Switzerland
Reading requires linking letter strings to already well-established auditory speech representations (Karipidis et al., 2021). During reading acquisition, this mapping is supported by phonological awareness, defined as the ability to detect and manipulate the sound structure of speech. Audiovisual (AV) integration and phonological awareness are closely related (Oron et al., 2016) and crucial prerequisites for reading development (Melby-Lervåg et al., 2012). In proficient adult readers, it remains unclear whether phonological awareness is specifically related to AV processing or more generally to unimodal language processing, and whether AV integration generalizes beyond letter–speech sound mapping to sentence-level processing. The present pilot study thus examined AV integration at the sentence level and the relationship between phonological awareness and unimodal versus audiovisual sentence processing. 20 German-speaking, typically reading adults (N = 20; 16 female; mean age = 26.9 years, SD = 2.8) performed two runs of an fMRI dynamic audiovisual sentence localizer followed by a behavioral phonological awareness task. The localizer used a block design with nine conditions, including auditory, visual, matching AV, and mismatching AV sentences, and corresponding control stimuli (spectrally-rotated noise-vocoded speech, checkerboards). Whole-brain and ROI analyses compared activation across modalities (auditory, visual, and audiovisual sentences vs. control). ROI analyses extracted mean beta values from 8 mm spheres centered on Neurosynth-derived phonological regions: left inferior frontal gyrus (IFG), inferior temporal and supramarginal gyri, bilateral superior temporal gyri (STG). AV integration was investigated using supra-additivity (AV > A + V), conjunction (AV > A ∩ AV > V), and congruency (match ≠ mismatch) contrasts. Phonological awareness was measured using a German spoonerism task requiring the exchange of the first phonemes. Preliminary analyses of whole-brain AV integration contrasts did not yield significant clusters (pCDT < .001, pFWEc<.05). Only audiovisual mismatch (mismatch > match) elicited activation in the middle and superior temporal gyri, left IFG, and calcarine cortex. Whole-brain analyses revealed a shared language network across modalities (IFG, precentral, temporal regions), with auditory processing mostly engaging superior temporal gyri, visual processing recruiting occipitotemporal regions, and AV processing combining both. ROI analyses revealed a selective positive correlation between phonological awareness and activation in the left IFG during AV sentence processing (r = 0.63, pFDR = 0.042), while no significant associations were found for unimodal auditory or visual sentence processing. The absence of classical AV integration effects may suggest that sentence-level processing relies more on higher level linguistic integration, such as the cross-modal combination of meaning and syntax, rather than on low level sensory binding. However, sensitivity to mismatch was reflected by IFG activation, consistent with its role in processing incongruent stimuli (Gao et al., 2023). Activation in this region during AV processing was also selectively associated with phonological awareness, consistent with prior work implicating the left IFG in phonological awareness in both children and adults (Hernandez et al., 2013; Turkeltaub et al., 2003). These preliminary findings indicate the importance of the left IFG during AV sentence processing and suggest that phonological awareness is particularly related to cross-modal processing demands, rather than unimodal sentence processing.
Topic Areas: Multisensory or Sensorimotor Integration, Reading