Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Feasibility of synthesizing inner speech from electrocorticogram using an autoencoder-based unsupervised learning
Poster Session B, Wednesday, September 30, 4:30 - 6:30 pm, Wangari Maathai
Jihun Hwang1, Hongsang Lee1, Chun Kee Chung2, Chang-Hwan Im1; 1Hanyang University, 2Seoul National University
Recent advancements in speech brain-computer interfaces (BCI) have demonstrated the feasibility of decoding overt and attempted speech using electrocorticography (ECoG) from the sensorimotor cortex (SMC) [1]. However, decoding inner speech remains a significant challenge; inner speech has relatively weaker neural signals. To overcome this limitation, we propose a novel approach utilizing unsupervised learning based on an autoencoder. By extracting latent feature vectors from ECoG signals recorded during inner speech and subsequently applying a supervised learning scheme for reconstruction, we successfully achieved the synthesis of audible speeches from ECoG signals. Four subjects participated in this study under IRB approval at Seoul National University Hospital (IRB No. 2011-087-1173). The experimental paradigm involved overt, mimed, and inner speech tasks using 108 words across three sessions. ECoG signals were recorded from the SMC at 2,000 Hz, while speech audio was simultaneously captured at 48 kHz. ECoG data underwent high-pass filtering and common average referencing (CAR), after which high-gamma envelopes (70–150 Hz) were extracted via the Hilbert transform and downsampled to 200 Hz. Corresponding speech audio was downsampled to 16 kHz and converted into 13-dimensional Mel-frequency cepstral coefficients (MFCCs) with cepstral mean normalization applied. The proposed framework utilizes an unsupervised autoencoder for feature extraction, consisting of an encoder with two 32-unit bidirectional LSTM (bLSTM) layers and a linear layer projecting inputs to a 4-dimensional latent vector, alongside a symmetric decoder. For subsequent supervised speech synthesis, a regression model with three 16-unit bidirectional gated recurrent unit (bGRU) layers followed by a linear output layer mapped the latent neural features to the target acoustic features. The model's synthesis performance was evaluated using a leave-one-trial-out cross-validation scheme. Visual and acoustic comparisons between the Mel spectrograms of the original overt speech utterances and the corresponding spectrograms synthesized from inner speech ECoG signals demonstrated highly consistent spectral structures. Quantitative analysis across all four participants revealed that the proposed autoencoder-based model achieved a significantly higher average Pearson correlation coefficient (PCC) of 0.85 ± 0.072 compared to the baseline direct synthesis model, which yielded a PCC of 0.84 ± 0.080 (p < 0.05, FDR-corrected). This statistically significant improvement indicates that the latent feature spaces successfully isolated robust speech-related representations. In this study, we demonstrated the feasibility of synthesizing unseen inner speech from ECoG signals by employing an autoencoder-based feature extraction model. The unsupervised learning stage proved critical in overcoming the characteristically low signal-to-noise ratio of inner speech by compacting complex cortical activations into a robust 4-dimensional latent space. This approach effectively addresses the long-standing vocabulary limitation in speech BCIs without relying on invasive single-unit recordings. Consequently, these findings establish a promising foundation for the future development of vocabulary-generalized speech neuroprostheses designed to restore naturalistic, real-time communication for individuals with severe speech and motor impairments.
Topic Areas: Computational Approaches,