In this article, the authors apparently "reconstruct" images from brain activity represented by functional MRI. I'm impressed, but I don't really understand what is being done.

I would like to understand their experimental setup (not the model itself) especially how the model is trained: what are the instances that the model is presented with, what do they consist of? Does the model maps a MRI directly to the corresponding picture??

(note that I'm not knowledgeable about computer vision)


1 Answer 1


The training data consists of fMRI images recorded while the subjects were looking at a set of natural scenes and the objective is to train a model to regenerate the natural scene from the fMRI images. The images also have some text annotations (an average of 5 per image). The study uses the fMRI images from four subjects.

The fMRI images are used in two ways. Firstly, fMRI data for the early visual cortex is used to predict a latent representation of the image. Secondly, fMRI data for the higher (ventral) visual cortex is used to predict latent text representations. Then these latent representations are combined and decoded to generate the reconstructed image.

  • $\begingroup$ Thank you for your answer. I'm still not clear about whether the model is shown the original pictures during training? And is it shown pairs of fMRI + corresponding picture? Doesn't the decoding stage requires the model to know how to associate a latent representation with a picture? $\endgroup$
    – Erwan
    Mar 19, 2023 at 12:42
  • 1
    $\begingroup$ @Erwan - They only train "linear models that map fMRI signals to each LDM component" and don't fine tune the LDM, so the model already knows how to generate a picture from the latent representation. I haven't found anything in the paper that describes how they construct the targets for training these linear models. This part is unclear to me as well, but I assume the pictures were used during this process. $\endgroup$
    – Lynn
    Mar 19, 2023 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.