Autoencoder: How should hidden layer be used?

Question

I'm building a variational autoencoder to generate faces. I'm using gray-scale images with the size 30x30. I started with a very simple model:

Input Layer, 900 nodes, values 0-1

Latent Space, 10 nodes (5 for mean, 5 for variance)

Output Layer, 900 nodes

The inputs of the latent space layer are normalized with Batch Normalisation and I use the sigmoid function on its outputs.

The model works okay and the input in the latent space layer controls things like background color. It produces images like these ones:

My problem is that whenever I add a hidden layer to the decoder and encoder, the images produced by the decoder will have the same shape, no matter what I input. When I change the input there is only noise that gets added. Here is are some image after I added the hidden layers:

The model was like this:

Input Layer, 900 nodes, values 0-1

Hidden Layer, 10 nodes

Latent Space, 10 nodes (5 for mean, 5 for variance)

Hidden Layer, 10 nodes

Output Layer, 900 nodes

The inputs of every layer are normalized with Batch Normalisation and I use the sigmoid function on the output of all layers apart from the input and the output layer.

Has someone an idea what I could change or what I'm doing wrong?

Answer 1

You may be facing the "posterior collapse" problem, where the model ends up ignoring the latent variables, which become uninformative.

To understand the problem, you can check the articles:

• Understanding Posterior Collapse in Generative Latent Variable Models (ICLR workshop 2019)
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse(ICML'2020)
• Don’t Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (NeurIPS'2019).

One way to avoid it are $\delta$-VAEs (see Preventing Posterior Collapse with delta-VAEs).