Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Cortical tracking of the multi-layered structure of sign language
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Joaquin Ordoñez1,2, Chiara Luna Rivolta1, Jaimy Hannah4, Mikel Lizarazu1, Giovanni Di Liberto4, Brendan Costello1,3; 1Basque Center on Cognition, Brain and Language, 2University of the Basque Country, 3Ikerbasque, 4Trinity College Dublin
Sign languages are conveyed through a complex visual signal produced by multiple articulators moving simultaneously, including the hands, head, torso, and face. Manual articulators (i.e., the two hands) primarily convey lexical and phonological information, while non-manual articulators contribute prosodic, syntactic, and discourse-level structure (Sandler, 2018). Sign language comprehension therefore requires the integration of multiple sources of information into coherent linguistic representations. Previous studies have investigated how the brain tracks the sign language signal, showing that neural activity synchronizes with the temporal dynamics of sign language in slow frequency bands. However, previous research has not fully captured the multi-layered structure of sign language, either by relying on global measures of visual change (Brookshire et al., 2017; 2021) or by analyzing the neural tracking of individual articulators independently (Rivolta et al., 2025). As a result, it remains unclear how neural activity simultaneously tracks the coordinated dynamics of multiple articulators during sign language comprehension. The present study addresses this limitation by applying multivariate Temporal Response Function (mTRF) modelling to MEG recordings from 27 deaf signers of Spanish Sign Language (LSE) viewing one-minute narratives in LSE with a probe task to ensure attention. Unlike coherence-based approaches, mTRF modelling enables the simultaneous inclusion of multiple articulator predictors within a single encoding model, allowing neural responses to be modelled as a function of their combined kinematic dynamics across time lags (Crosse et al., 2016). This approach makes it possible to estimate the extent to which individual articulators explain neural variance while taking the contribution of other articulators into account. To obtain the kinematic pattern of each articulator, we used a depth-based tracking system to track the positions of multiple body points in three dimensions. From these recordings, we extracted speed time series for the dominant hand, non-dominant hand, head, torso, and a non-linguistically relevant control point (hip). A full mTRF model including all articulator predictors will be compared against reduced models, each excluding an individual articulator. The strength of neural tracking will be quantified as the correlation between recorded and predicted neural activity and averaged across sensors to obtain a subject-level prediction accuracy measure. Statistical comparisons between full and reduced models will be used to determine whether excluding a given articulator significantly reduces prediction accuracy, thereby estimating its unique contribution to cortical tracking while accounting for the correlated dynamics of natural signing. We predict that the dominant and non-dominant hands will explain the greatest proportion of neural variance, with the dominant hand showing the strongest contribution due to its primary role in conveying lexical and phonological information. This study will help characterize how linguistic information distributed across multiple articulators is integrated into coherent linguistic representations during sign language comprehension, as well as the relative contribution of each articulator to this process.
Topic Areas: Signed Language and Gesture,