1
$\begingroup$

Posterior collapse means the variational distribution collapse towards the prior: $\exists i: s.t. \forall x: q_{\phi}(z_i|x) \approx p(z_i)$. $z$ becomes independent of $x$. We would like to avoid it when training VAE.

When maximizing the variational lower bound on the marginal log-likelihood, we would like to minimize the kl-divergence: $KL(q_\phi(z|x)||p(z))$. That is to keep the approximate posterior close to the prior. To have a tight bound, the $KL=0$.

Are these two not contradicting each other? In minimizing kl-divergence case, does $KL=0$ mean posterior collapse?

(I feel I am mixing up some concepts here but not sure what exactly.)

$\endgroup$

1 Answer 1

1
$\begingroup$

You are correct in your reasoning (though I dont' know if I'd call it a "collapse")- the "collapse" to the prior is the ideal intent of a VAE. As a VAE is a type of autoencoder, it's purpose is to learn a latent encoding space $Z$ from which you could, in principle, "draw" random points. An example is a VAE trained to generate images of faces, where each point $z \in Z$ represents a particular face. The encoder in a VAE is a way to transform points into that space, and the decoder is a way to transform them out of it.

In practice, however, we run into issues with data distributions, we may have data in one part of the latent space over-represented in some places, and under-represented in others. The "variational" part of VAEs adds regularization with this. Instead of encoding a single point in the latent space, it encodes a distribution over points. This helps mitigate this problem.

So rather than going from $$x \rightarrow z \in \mathbb{R}^d$$

a VAE encoder goes from $$x \rightarrow z\sim\mathcal{N}( \mu_x, \sigma_x)$$

What this means is that every point $z\in Z$ should be drawn from the prior distribution, $p(z)$. The encoder, $e(x)$ is just a means to find that spot $z$ in the encoding space (or, more specifically, it's distribution parameters).

The distribution $q(\cdot | x)$ is just an approximation of the distribution $P(z)$ that uses the datapoint $x$ to generate the distribution parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.