Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Deco-B: Brain-to-speech Decoding across Languages
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
This poster is part of the Sandbox Series.
Vincenzo Verbeni1, Li-Chuan Ku1, Ekain Arrieta2, Xabier De Zuazo2, Nicola Molinaro1,3; 1BCBL - Basque Center on Congition, Brain and Language, 2HiTZ Center, University of the Basque Country – UPV/EHU, Spain, 3Ikerbasque, Basque Foundation for Science, Bilbao, Spain
Driven by recent developments in artificial intelligence, brain-to-speech decoding technologies are rapidly progressing. Nonetheless, decoding paradigms have been tested almost exclusively on monolingual data, overlooking the reality that most humans can speak more than one language. Assessing the performance of neural decoding models on bilingual data is a critical next step: the organization of meaning in bilingual brains could indeed be leveraged to enhance the performance of future brain-computer interfaces. The present study reports preliminary findings from Deco-B, an ongoing project aimed at decoding continuous speech from magnetoencephalography (MEG) signals in Basque-Spanish bilingual individuals. Our goal is to assess whether shared semantic representations in the bilingual brain can support cross-language decoding – the reconstruction of speech in one language (e.g., Basque) with a model trained exclusively on brain responses elicited by the other (Spanish). To this aim, we are collecting over 60 hours of MEG data across 10 participants listening to 24 auditory narratives (12 Basque-Spanish translation pairs). The data will serve as input to our brain-to-speech model, which follows an encoding-decoding architecture. During the encoding phase, a linear model is trained to map word-level embeddings (extracted from narratives in one language) to the associated neural activity. Embeddings are extracted via large language models (LLMs) and capture cross-lingual semantic properties, making them suitable for transfer across languages. In the decoding phase, a generative LLM – provided with minimal context in the target language – produces candidate word continuations. Each candidate’s embedding is mapped onto the original encoding space to reconstruct the expected MEG signal, which is then matched against the recorded response to the target language: based on the similarity between the two, the best candidate is fed back into the generative model, enabling the progressive decoding of the target narrative. Within this framework, decoding accuracy is contingent on the overlap between neural responses to translation-equivalent stimuli: provided that embeddings capture language-agnostic semantic properties, decoding performance should be highest when responses to translation pairs converge. Therefore, a necessary precondition is that embeddings reliably track semantic activations in our neural data. As a first validation step, we fitted ridge regression encoding models mapping word embeddings extracted via Latxa 7B (a Basque-Spanish LLM) to single-subject MEG responses epoched around word onsets (−200 to 1000 ms). Leave-one-story-out cross-validation revealed that embeddings significantly predict MEG activity in both languages: encoding effects concentrated at mid-to-late latencies, consistent with the temporal profile of semantic integration processes (N400; late positivity). Building on these findings, we are currently implementing the full decoding pipeline. We anticipate that cross-language decoding accuracy will be modulated by two factors: stimulus familiarity – with higher accuracy expected for narratives present in the training set – and spatial specificity, with source-localized activity from canonical semantic hubs (anterior temporal lobe, angular gyrus, inferior frontal gyrus) predicted to outperform data from non-language related regions. If successful, our pipeline can be employed to develop decoding systems that generalize to multiple languages without requiring separate neural recordings for each – a significant advance in scalability and clinical applicability.
Topic Areas: Computational Approaches, Multilingualism