# Does minimizing kl divergence (i.e. keep approximate posterior close to prior) contradict the goal of avoiding posterior collapse?

Posterior collapse means the variational distribution collapse towards the prior: $$\exists i: s.t. \forall x: q_{\phi}(z_i|x) \approx p(z_i)$$. $$z$$ becomes independent of $$x$$. We would like to avoid it when training VAE.

When maximizing the variational lower bound on the marginal log-likelihood, we would like to minimize the kl-divergence: $$KL(q_\phi(z|x)||p(z))$$. That is to keep the approximate posterior close to the prior. To have a tight bound, the $$KL=0$$.

Are these two not contradicting each other? In minimizing kl-divergence case, does $$KL=0$$ mean posterior collapse?

(I feel I am mixing up some concepts here but not sure what exactly.)

## 1 Answer

You are correct in your reasoning (though I dont' know if I'd call it a "collapse")- the "collapse" to the prior is the ideal intent of a VAE. As a VAE is a type of autoencoder, it's purpose is to learn a latent encoding space $$Z$$ from which you could, in principle, "draw" random points. An example is a VAE trained to generate images of faces, where each point $$z \in Z$$ represents a particular face. The encoder in a VAE is a way to transform points into that space, and the decoder is a way to transform them out of it.

In practice, however, we run into issues with data distributions, we may have data in one part of the latent space over-represented in some places, and under-represented in others. The "variational" part of VAEs adds regularization with this. Instead of encoding a single point in the latent space, it encodes a distribution over points. This helps mitigate this problem.

So rather than going from $$x \rightarrow z \in \mathbb{R}^d$$

a VAE encoder goes from $$x \rightarrow z\sim\mathcal{N}( \mu_x, \sigma_x)$$

What this means is that every point $$z\in Z$$ should be drawn from the prior distribution, $$p(z)$$. The encoder, $$e(x)$$ is just a means to find that spot $$z$$ in the encoding space (or, more specifically, it's distribution parameters).

The distribution $$q(\cdot | x)$$ is just an approximation of the distribution $$P(z)$$ that uses the datapoint $$x$$ to generate the distribution parameters.