Release of CortexMAE and Brainmarks

📄 Paper 💻 Code 🤗 Model 📊 Benchmarks

TL;DR. Today we're releasing CortexMAE: a family of fMRI foundation models trained on 2.1K hours of open fMRI data. We're also releasing Brainmarks: an open benchmark suite for evaluating fMRI foundation models. The full paper is on arXiv (accepted to ICML 2026), our code is on GitHub, and model checkpoints are on Hugging Face.

At Sophont we really like fMRI. Honestly, it seems a bit like magic. Somehow we are able to construct detailed 3D pictures of what's going on inside a person's brain, using only magnets. At the same time, the data have many serious challenges (low temporal resolution, incredibly low SNR, loose coupling with neural firing, etc.). One of our main goals is to build models that can extract clinically useful signals from these remarkable, but also very challenging data.

Today, we're announcing our next step toward this goal. We're releasing CortexMAE: a family of fMRI foundation models trained on 2.1K hours of fMRI data from the Human Connectome Project using the Masked Autoencoder paradigm. To promote reproducible benchmarking, we also release Brainmarks: an open evaluation suite for fMRI foundation models, inspired by another Sophont project Medmarks.

We release a family of CortexMAE models trained with different fMRI data representations: our flagship model based on cortical flat maps, plus efficient parcellation-based models and dense volume-based models.
We release >50 additional checkpoints covering data scale, model scale, alternative parcellations, etc. All checkpoints include the original configs and logs for reproducibility.
Our Brainmarks evaluation suite includes curated pretraining and benchmark datasets available in convenient webdataset and Arrow formats, plus code for reproducible evaluation.
Our models achieve SotA performance among all fMRI foundation models at decoding individuals' dynamic mental state.

Figure 1. CortexMAE on fMRI flat maps. Example is from the Natural Scenes Dataset (NSD), not seen during training. Credit to pycortex for creating the flat map and to Florian Rupprecht for the animation.

CortexMAE: a new fMRI foundation model family

In order to directly adapt the Masked Autoencoder (MAE) framework to fMRI, our idea was to convert the 3D fMRI volumes into 2D images using a standard cortical flat map projection. (Specifically, we use the very nice flat map from pycortex.)

After converting the data to flat maps, we can apply vanilla MAE to fMRI with basically zero changes. The inputs are fMRI flat map “videos” with 90% of patches masked. The model is trained to predict the pixel values for the unobserved patches. Our default model is a ViT-B/16 trained for one day on a single H100.

How do you “tokenize” fMRI data?

CortexMAE builds on a growing body of fMRI foundation model work. One of our main contributions is to explore the question: How exactly should fMRI FMs “tokenize” 3D fMRI volumes? All prior models use one of two approaches:

Parcellation methods reduce each 3D fMRI volume to a list of ~400 numbers by averaging within a fixed set of brain “parcels”.
Volume methods don't do any reduction and just model the native 3D data directly.

We hypothesize that we may be missing a “Goldilocks zone” of fMRI representations that are more information-rich than parcellation, yet more structured and efficient than volume. We propose cortical flat maps as an initial attempt to fill this gap.

**Figure 2.** Spectrum of fMRI data representations and their corresponding ViT patches. Parcellation “patches” are single scalars, flat map patches are 2D squares, and volume patches are 3D cubes.

To test this hypothesis, we train variants of the CortexMAE model using all three approaches, resulting in a multi-representation family of fMRI foundation models:

CortexMAE-P: an efficient parcellation-based model
CortexMAE-F: our flagship model based on flat maps
CortexMAE-V: a dense volume-based model

There are many other possibilities beyond these three. We hope to see future work explore even more effective ways of representing fMRI data.

Brainmarks: an open reproducible evaluation suite

Competing on shared benchmarks is often what drives progress. Although prior works evaluate on similar source datasets (HCP, ADNI, ABIDE, etc.), they use different data curation and evaluation protocols. To make it easier for future work to compare against ours, we assembled our curated benchmark datasets and evaluation code into an open, reproducible evaluation suite that we call Brainmarks.

Brainmarks currently includes >30 benchmark tasks across 7 publicly available source datasets. In our paper, we focus on a core group of 8 benchmarks.

Benchmark	Classes	Subjects	Samples	Duration	TR
ABIDE Dx	2	826	826	300s	2.0s
ADHD200 Dx	2	430	430	300s	2.0s
ADNI Dx	2	410	410	300s	3.0s
PPMI Dx	2	662	662	300s	2.5s
HCP-A Age	4	560	560	350s	0.7s
HCP-A Sex	2	584	584	350s	0.7s
HCP-YA Task21	21	614	28K	16s	1.0s
NSD COCO24	24	8	49K	16s	1.0s

These include 6 commonly used benchmarks for predicting subject-level traits, such as age or clinical diagnosis, as well as 2 benchmarks for decoding dynamic brain state: HCP-YA Task21 cognitive state decoding and NSD COCO24 image category decoding.

Brainmarks currently supports 7 fMRI foundation models (including CortexMAE). We hope to receive PRs for new models that achieve new SotA!

Main results

Denoising fMRI by masked reconstruction

Here we visualize our model's predictions. For a given input (left), we compute a full data reconstruction (right) by averaging the masked predictions for 100 random masks.

Figure 3. Denoising fMRI data with CortexMAE. Original data are dominated by high frequency noise, while reconstructions are smooth and structured. Model reconstructions are computed by averaging masked predictions over 100 random masks (mask ratio 90%). Example is from NSD, not seen during training.

Whereas the input data are noisy, the model's reconstructions are structured and spatiotemporally smooth. There are also lots of interesting dynamics that are obvious in the reconstruction but difficult to see in the original data (default mode network oscillation, traveling waves in visual cortex, large activation of the sensorimotor hand areas).

Importantly, when the model makes a prediction for any given patch, the ground truth pixel values are hidden. This leads to a denoised prediction: structured signal that is predictable from context is preserved, while unstructured noise is left behind.

At the same time, it's likely that the predictions are too smooth. The model has no way of predicting fine-grained details that are independent of the context. We hope that future work can improve on this!

CortexMAE achieves SotA on brain state decoding

The unique value of functional MRI is its ability to capture dynamic patterns of brain activity. CortexMAE excels at representing these dynamic patterns. On downstream probe benchmarks testing how well fMRI FM representations can support cross-subject mental state decoding, our parcellation- and volume-based models (CortexMAE-{P,V}) are best in their respective classes, while the flagship flat map model (CortexMAE-F) achieves overall SotA (supporting our “Goldilocks zone” hypothesis).

**Figure 4.** fMRI foundation models achieve strong performance on cross-subject dynamic cognitive state prediction. The CortexMAE family achieves current SotA. Evaluation uses attentive probe.

But importantly, this cross-subject decoding performance is much worse than what you can get by training subject-specific decoding models (>60% accuracy on NSD COCO24 image category decoding). There is a lot of room to explore how to adapt the current “population-level” fMRI FMs to take advantage of this fine-grained subject-specific information.

All fMRI foundation models struggle at clinical diagnosis

The ultimate goal of an fMRI foundation model is to help understand, diagnose, and treat brain and mental health conditions. A first step is to test whether models can distinguish between individuals with a clinical diagnosis (autism, Alzheimer's, etc.) and control subjects, solely using fMRI.

Surprisingly, we find that all fMRI foundation models (including our own) have relatively weak performance in this setting. Classification accuracy is inconsistent across models and datasets, and poorly differentiated from a simple baseline that uses hand-crafted functional connectivity features. (In contrast, the best models on state decoding are >20σ above baseline.)

**Figure 5.** fMRI foundation models struggle on clinical diagnosis prediction. Evaluation uses linear probe averaged over 100 random train/test splits.

In order to clearly establish the clinical potential of fMRI foundation models, we think more informative benchmarks are needed. This means putting more effort into quality control for the current standard benchmarks, as well as building new benchmarks. One idea is to explore predicting clinically relevant changes in brain state (e.g. treatment response), rather than static traits like diagnosis category.

Scaling with data and compute

A crucial test of any foundation model approach is: Does it scale? Our answer is: yes, but we are limited by data. Masked reconstruction performance scales with increasing dataset size according to a strict power law.

**Figure 6.** Masked fMRI modeling scales with dataset size.

As we scale model size, however, performance saturates around 37M parameters. In the paper, we also show how even the strong data scaling trend becomes weaker when we test on out-of-distribution data from another dataset. These results suggest that fMRI foundation models need more data, and especially more diverse data.

**Figure 7.** Masked fMRI modeling scales with model size, but only up to ~37M parameters.

We should also recognize that fMRI “foundation models” still live in the toy regime (<100M parameters, <10B tokens) compared to modern vision and language models. There is a lot of room to scale these models up, if we can find the right data for it.

Conclusion

There are many more results in the main paper, check it out!

Looking ahead, we feel that the field of fMRI foundation models, and foundation models for neuro data more broadly, is still in its early stages. Our work leaves many future directions open. We are currently working on building multimodal MRI foundation models of brain structure as well as function. We are participating in the FOMO26 MRI foundation model challenge and our team is open to new contributors. Find us in the MedARC Discord!

We are also interested in working with academic researchers and commercial partners to build applications of our models . We made the models open so you can test them out yourself. To get started, check out our GitHub! Send us a message if you're interested in discussing more.

Acknowledgements

Thanks to fal for providing compute that supported this research. Thanks to MedARC contributors Melvin Selim Atay, Mohammed Baharoon, Atmadeep Banerjee, Uday Bondi, Pierre Chambon, Alexey Kudrinsky, Souvik Mandal, Ashutosh Narang, Alex Nguyen, Yashvir Sabharwal, Kevin Son, and Dingli Yu for contributing to an earlier version of this project. Thanks to Zijiao Chen, Gregory Kiar, and Florian Rupprecht for helpful discussions on an earlier version of this work. Thanks to the anonymous reviewers for helpful feedback. Thanks to the MedARC Discord community for being the public forum from which this research was developed.

Citation

Please cite this work as:

@inproceedings{lane2026scaling,
    title={Scaling Vision Transformers for Functional MRI with Flat Maps},
    author={Connor Lane and Mihir Tripathy and Leema Krishna Murali and Ratna Sagari Grandhi and Shamus Sim Zi Yang and Sam Gijsen and Debojyoti Das and Manish Ram and Utkarsh Kumar Singh and Cesar Kadir Torrico Villanueva and Yuxiang Wei and Will Beddow and Gianfranco Cortés and Suin Cho and Daniel Z. Kaplan and Benjamin Warner and Tanishq Mathew Abraham and Paul S. Scotti},
    booktitle={ICML},
    year={2026},
}