Search Abstracts | Symposia | Slide Sessions | Poster Sessions
RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain
Poster Session E, Friday, October 2, 11:00 am - 1:00 pm, Wangari Maathai
Omer Moussa1, Mariya Toneva1; 1Max Planck Institute for Software Systems (MPI-SWS)
(1) Introduction: A foundation model of language-evoked brain activity could enable scalable, population-level neuroscience — predicting cortical responses to natural speech in new participants and datasets without refitting a model per person. Such a model must satisfy two desiderata: (i) accurate zero-shot prediction in unseen participants, and (ii) rapid few-shot improvement when a small amount of a person's fMRI is available. These goals come from the fact that responses are consistent across listeners in early auditory cortex but change sharply in higher-order language regions. Existing methods address only one side: voxel-wise encoders are fundamentally per-participant; current foundation-style encoders predict at the group level but adapt poorly to new individuals. Here, we introduce RABBiT, an encoder that meets both desiderata and recovers canonical music vs speech localizers without retraining. (2) Methods: RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning) is designed to support both accurate zero-shot prediction of speech-evoked responses and efficient few-shot personalization to new subjects. It does this through modeling brain regions with a Shared--Idiosyncratic Decomposition (SID): a shared component that captures reliable structure across participants, while a participant-specific deviation captures individual responses. The shared pathway supports zero-shot prediction for any speech stimulus, and the idiosyncratic pathway enables rapid personalization without fitting a new high-dimensional voxel-wise model for every participant. To learn these components from speech, RABBiT first adapts a pretrained speech model directly to language-evoked fMRI activity. A Temporal Brain Transformer (TBT) then learns how different brain regions selectively integrate information over the speech output sequence, replacing fixed temporal pooling and voxel-wise processing with explicit region-level representations. SID finally maps these representations to high-resolution cortical activity. The model is trained end-to-end on the CNeuroMod Friends dataset (~515K TRs). (3) Results: Zero-shot, RABBiT reaches the leave-one-out inter-subject consistency estimate (ISC) on 324 held-out subjects, outperforming both a linear baseline and the much larger TRIBEv2 foundation model. Across bilateral language regions, RABBiT achieves group-level predictions approaching the ISC despite being trained on only six participants. The model also reproduces the distinct cortical pathways for music and speech across held-out datasets, indicating that the learned representations generalize beyond the training cohort and can be used as a foundation for neuroscientific studies. The largest zero-shot gains occur in auditory and temporal language regions, including STG/STS and temporal pole. Few-shot, updating only the SID idiosyncratic pathway (~115K parameters) improves over zero-shot with just 10 minutes of calibration and outperforms voxel-wise ridge regression (p<0.001), while using roughly three orders of magnitude fewer trainable parameters. The strongest gains (30--90\%) occur in higher-order language regions including IFG, angular gyrus, and supramarginal gyrus---the most idiosyncratic regions. Finally, the learned region queries expose interpretable structure: a greedy walk through query space recovers a coherent trajectory from auditory cortex toward frontal language regions without spatial or hierarchy supervision. (4) Conclusion: RABBiT shows that accurate personalization of language-evoked brain activity does not require fitting a new high-dimensional model for every participant. Instead, separating prediction into shared and participant-specific components supports both strong zero-shot inference and rapid few-shot adaptation.
Topic Areas: Methods, Computational Approaches