What kind of neural network structure is suitable for image to image learning?

Question

There exists a mapping from input image to out image. Say input image is a piece of paper with a square hole in the center, and output image is the shadow of the input image when light shines on the paper, which also looks like a square, but with some blurs and different in size. (The real pattern is far more complicated than this square hole) I'm trying to implement a neural network that can learning this kind of mapping, when given the input image, the network can predict the "shadow" of the input image. I've tried to use a convolutional autoencoder-like structure (like this https://blog.keras.io/building-autoencoders-in-keras.html) by replace the target image from the input image it self to the output image training set. But it didn't looks work well. I would like to know what kind of neural network is suitable for this task? And what kind of loss function is suitable?

Answer 1

I would like to know what kind of neural network is suitable for this task?

A convolutional autoencoder is a good thing to try, but there is a minor issue with it for this type of problem: the latent representation has often lost spatial awareness (especially if FC layers were used at the end of the encoder). For a task like this, a pixel $p_{ij}$ in the input is likely to be far more related (and useful) to the corresponding (same position) pixel $\widehat{p}_{ij}$ in the output, compared to some other, far away pixel $p_{\ell k}$ in the input.

For this reason, I'd suggest using a U-net architecture, as this preserves spatial localities throughout the network.

References:

• Long et al, Fully Convolutional Networks for Semantic Segmentation

• Ronneberger et al, U-Net: Convolutional Networks for Biomedical Image Segmentation

And what kind of loss function is suitable?

A good starting point is the standard mean $L_1$ or $L_2$ loss over the output image. Since your images are not natural images, I hesitate to recommend perceptual losses, but they could work as well.

Without more details about your architecture and data, I'm afraid I cannot say much more than these though.