Search Abstracts | Symposia | Slide Sessions | Poster Sessions
Distinguishing passive listening and inner speech from electrocorticogram using a multi-frequency convolutional neural network
Poster Session C, Thursday, October 1, 10:45 am - 12:45 pm, Wangari Maathai
Gaeun Lee1, Jihun Hwang1, Chunkee Chung2, Changhwan Im1; 1Hanyang University, 2Seoul National University
In practical speech brain-computer interfaces (BCIs), distinguishing between inner speech and passive listening is critical, as both states share overlapping neural substrates, introducing a substantial risk of false activations [1]. To address this challenge, we propose MFNet (Multi-Frequency Convolutional Neural Network), a CNN-based model for differentiating passive listening from inner speech states from electrocorticography (ECoG). One subject participated in the study, completing 10 sessions (including one repeated session) of passive listening and inner speech tasks with 270 different sentences. ECoG signals from the sensorimotor cortex (SMC) and auditory cortex were acquired via the Neuvo system at 2 kHz, while audio stimuli for the passive listening task were presented at 48 kHz. The ECoG data underwent high-pass filtering for detrending using a 4th-order zero-phase Butterworth filter with a cutoff frequency of 0.5 Hz, followed by common average referencing (CAR) and notch filtering at 60, 120, and 180 Hz using 2nd-order IIR notch filters to suppress line noise. Analytic envelopes spanning theta (4-8 Hz) through high-gamma band (70-150 Hz) were extracted via 8th-order Butterworth bandpass filtering and the Hilbert transform. All features were downsampled to 200 Hz and z-score normalized per channel across the full session. This study was approved by the Institutional Review Board at Seoul National University Hospital, Republic of Korea (IRB No. 2011-087-1173). The proposed MFNet takes multi-frequency ECoG envelopes as input and extracts temporal, spatial, and cross-frequency features through three successive convolutional blocks, followed by temporal mean pooling and a linear classifier. The model was trained using a 10-fold cross-validation with AdamW optimizer (learning rate = 10-4), cross-entropy loss, and data augmentation based on Gaussian noise addition and channel dropout, with the epoch yielding the highest mean validation accuracy across folds selected as the final model. Final predictions were evaluated under three strategies: center crop (single centered window per trial), independent windows (per-window prediction without aggregation), and soft voting (probability averaging over 50% overlapping windows). Both training and classification were performed using 18 electrodes from the SMC and auditory cortex, selected for their relevance to speech motor and auditory processing, respectively. Among three validation strategies tested—center crop, independent windows, and soft voting—the soft voting approach achieved the highest mean classification accuracy of 83.83 ± 3.77%, outperforming the center crop (76.00 ± 3.87%) and independent window (68.03 ± 2.97%) strategies. We demonstrated the feasibility of distinguishing passive listening and inner speech from ECoG signals using a CNN-based network architecture. The observed differences across validation strategies may reflect trial-by-trial variability in the timing and spatial distribution of discriminative neural signals, suggesting that ensemble-based approaches such as soft voting, which aggregate predictions across multiple temporal windows, are better suited to capture these inconsistencies than fixed-window methods. Additional analyses with more numbers of subjects will be presented at the conference. [1] Lu, et al., “Common and distinct neural representations of imagined and perceived speech,” Cereb. Cortex, vol. 33, pp. 6486–6493, 2023.
Topic Areas: Computational Approaches,