2
$\begingroup$

In the following work the authors apply a convolutional recurrent neural network (RNN) to predict the spatiotemporal evolution of microstructure represented by 2D image sequences. In particular, they apply some sort of 3D-CNN & LSTM architecture to predict crystal growth:

Crystal growth evolution

In the above picture, we can see RNN predictions (P) versus ground truth (G) from a testing case, in which the RNN outputs 50 frames based on 10 input frames.

Now, this is something new to me: how is possible for a RCNN to generate images as output? From my (limited) knowledge, the only structures able to generate new images in output are generative adversarial networks like GANs and Convolutional Encoder-Decoder NN (like VAE), but apparently the authors achieve these results by solely stacking together 3D-Convs and RNN units.

Have you ever seen these kind of architectures?

$\endgroup$

1 Answer 1

3
$\begingroup$

The authors provide this image in their supplemental information: enter image description here

There, you can see their explanation. The convolutional layers encode the image into some latent space representation. The RNN operates in this latent space, generating a new latent space representation based on the previous observations. For any latent space representation, the decoder can convert it into an image.

Thus the RCNN uses essentially the same procedure as the model types you mentioned (GANs, Convolutional Encoder-Decoders); there is a decoder that takes representations from the latent space to the image space.

$\endgroup$
1
  • $\begingroup$ I completely overlooked that! So it seems that they do a sort of image to image translation in the same setting of autoencoders. Interesting to use in between a RNN to take into account the time series. $\endgroup$
    – Betelgeuse
    Jan 18, 2022 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.