Brain Decoding and Reconstruction of concepts of visual stimuli from fMRI through deep diffusion models
Matteo Ferrante, Tommaso Boccato, Nicola Toschi · Proceedings on CD-ROM - International Society for Magnetic Resonance in Medicine. Scientific Meeting and Exhibition/Proceedings of the International Society for Magnetic Resonance in Medicine, Scientific Meeting and Exhibition · 2024
In vision, the brain is a feature extractor which works from images. We hypothesize that fMRI can mimic the latent space of a classifier, and employ deep diffusion models with BOLD data from the occipital cortex to generate images which are plausible and semantically close to the visual stimuli administered during fMRI. To this end, we mapped BOLD signals onto the latent space of a pretrained classifier and used its gradients to condition a generative model to reconstruct images. The semantic fidelity of our BOLD response to visual stimulus reconstruction model is superior to the state of the art.