Search Abstracts | Symposia | Slide Sessions | Poster Sessions
A Computational Framework for Sub-Letter Orthography
Poster Session D, Thursday, October 1, 4:30 - 6:30 pm, Wangari Maathai
Jack E. Taylor1,2,3, Christian J. Fiebach1,2; 1Department for Psychology, Goethe University Frankfurt, Frankfurt am Main, Germany, 2COBIC Brain Imaging Center, Goethe University Frankfurt, Frankfurt am Main, Germany, 3School of Psychology and Neuroscience, University of Glasgow, Glasgow, United Kingdom
Early, more visual stages of orthographic processing in visual word recognition (relating to shapes of letters/characters/words) are referred to in linguistics as “graphetic” (Meletis, 2020). These stages of processing have received less attention in the cognitive neuroscience of reading and word recognition than later stages (related to identities of letters/characters/words, letter order encoding, etc.). One reason for this is a lack of computationally explicit descriptions for graphetics and, thus also for the cognitive and neural representations of graphetics. Building on optimal transport theory, a field of mathematics concerned with identifying and describing mappings between distributions of mass that incur a minimum total cost, we propose a computational framework for describing graphetics. In support of this model, representational similarity analyses of data from two separate preregistered EEG studies show that optimal transport Wasserstein distances between letter shapes align strongly with neural activity, performing as well as artificial neural networks trained to classify letters. However, unlike artificial neural networks, the optimal transport framework additionally provides a more directly interpretable description of letter shape similarity. We further show that the optimal transport framework can provide insights into the emergence of invariance in letter recognition, via a distance measure that quantifies similarities in internal structures, invariant to a range of transformations: While the more retinotopic representations of letters captured by Wasserstein distance align better with earlier neural activity during the P100, the more invariant representations captured by Gromov-Wasserstein distance align better with later neural activity during and after the N170. The optimal transport framework additionally provides a means to interpolate between these perspectives; we conducted simulations suggesting that overall behaviour in letter recognition would be optimal in an intermediate “Goldilocks zone” of invariance. This is consistent with the expectation that complete invariance to affine transformations would be suboptimal because of letters that differ only by rotation and/or mirroring (e.g., b, d, p, q). Interestingly, however, our neural data suggest that there are instead two stages of processing with representations that reflect either complete retinotopy or complete invariance to such transformations, respectively. We also compare these results to data from 20 participants with dyslexia. Initial results suggest that readers with dyslexia may show reduced alignment with more invariant representations in later activity (post-N170). Finally, we present data from ongoing behavioural studies using the optimal transport framework to continuously interpolate between letter shapes. We show that (i) this makes it possible to model the “graphetic solution space” (Meletis, 2020): the space of letter shapes permissibly identified as a given letter. We further examine (ii) whether the perception of ambiguous letter shapes, produced by such optimal transport interpolations, can be biased, or “recalibrated” (Bertelson et al., 2003), if paired with unambiguous speech sounds. Overall, we provide empirical support for our proposal that an optimal transport framework provides a computationally explicit description of letter shapes that can be used to answer important and long neglected questions about the nature of cognitive and neural representations of the earliest, graphetic, stages of visual word recognition.
Topic Areas: Computational Approaches, Reading