Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Emojis and inferential processes: the neurophysiology of bridging in multimodal settings
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Gustavo Pimentel de Oliveira1, Marije Soto1; 1Federal University of Rio de Janeiro (Brazil)
This work investigates inferential bridging when it is triggered across different modalities of communication – specifically, between text and emojis, ubiquitous in everyday communication. We draw primarily on Grosz, Kaiser, and Pierini (2021), who show that emojis can, under certain conditions, behave as independent discourse units with anaphoric effect. We noticed that this anaphoric effect is frequently mediated through an inferential bridge, and asked whether emojis in multimodal anaphoric contexts could help clarify a classic question in comprehension research: how the dynamics of processing given and new information ("given-new" paradigm; Clark & Haviland, 1977) are reflected in neural processing. Bridging has been studied through event-related potentials (ERPs) in purely linguistic (Burkhardt, 2006) and purely visual (Cohn & Kutas, 2015) contexts, but only rarely in multimodal settings (see Zhang & Tsimpli, 2026, for an example focusing on modality switching). Following Burkhardt's design on definite-DP bridging, we constructed three experimental conditions varying in the informational dependency between text and image: in condition “given”, the emoji merely referred back to the previous context; in “bridged”, it triggered a bridging inference; in “new”, it bore no clear relation to the sentence. To prevent participants from forming positional expectations about the emoji, the experiment also included filler trials in which one or more sentence elements were replaced by emojis. Stimuli were validated through a web-based norming study (N = 241) using three 7-point Likert scales (visual clarity, coherence, information addition). The 120 most consistently rated stimuli (40 per condition) were presented via Rapid Serial Visual Presentation in the EEG experiment (N = 28), with ratings retained as continuous predictors for a regression-based ERP (rERP) analysis. In the categorical analysis (spatio-temporal permutation cluster test), we observed a classic N400 effect (a centroparietal negativity peaking at approximately 400ms) for bridging relative to given, and for new relative to given, with no significant difference for bridging relative to new. RERP analysis revealed that this negativity varied continuously with perceived coherence, where less coherent stimuli produced more negative responses. No P600 effect was observed. The presence of a graded N400 effect alongside the absence of a P600 distinguishes our findings from both Burkhardt's (2006) results for definite-DP bridging, where the two components were observed, and from Cohn and Kutas's (2015) results for visual-narrative bridging, where a P600 and a frontal negativity were observed but no classic N400. The pattern suggests that emoji-mediated discourse integration engages dependency-formation processes but not the integration cost typically indexed by the P600. We propose that this dissociation may reflect the structural independence of emojis as discourse units. In purely linguistic bridging, the target word is embedded within the sentence, preserving structural continuity even as informational dependency varies. Similarly, in Cohn and Kutas's visual narratives, the target panels are embedded within a structurally continuous sequence. Emojis, by contrast, constitute a modally and structurally distinct element – semantically dependent on the preceding text but not syntactically integrated into it. This structural discontinuity may modulate the integration process in ways that are not captured by the P600.
Topic Areas: Meaning: Discourse and Pragmatics,