I was going through this paper on Towards Text Generation with Adversarially Learned Neural Outlines and it states why the VAEs are hard to train for text generation due to this problem. The paper states

the model ends up relying solely on the auto-regressive properties of the decoder while ignoring the latent variables, which become uninformative.

please simplify and explain the problem in a lucid way.

  • $\begingroup$ Posterior collapse is best understood as an example of non-identifiability. If the posterior density of the latent variables equals its prior, it means there is not enough information in the sample to estimate the chosen model. For more information, see this paper by Wang et al. $\endgroup$
    – Durden
    Jun 26, 2023 at 20:23

1 Answer 1


With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks:

When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\phi}(z_d|x)=\mathcal{N}(\mu_d, \sigma^2_d)$, where $\mu_d$ and $\sigma_d$ are stable functions of input $x$. In other words, encoder distills useful information from $x$ into $\mu_d$ and $\sigma_d$.

We say a posterior is collapsing, when signal from input $x$ to posterior parameters is either too weak or too noisy, and as a result, decoder starts ignoring $z$ samples drawn from the posterior $q_{\phi}(z|x)$.

The too noisy signal means $\mu_d$ and $\sigma_d$ are unstable and thus sampled $z$'s are also unstable, which forces the decoder to ignore them. By "ignore" I mean: output of decoder $\hat{x}$ becomes almost independent of $z$, which in practice translates to producing some generic outputs $\hat{x}$ that are crude representatives of all seen $x$'s.

The too weak signal translates to $$q_{\phi}(z|x)\simeq q_{\phi}(z)=\mathcal{N}(a,b)$$ which means $\mu$ and $\sigma$ of posterior become almost disconnected from input $x$. In other words, $\mu$ and $\sigma$ collapse to constant values $a$, and $b$ channeling a weak (constant) signal from different inputs to decoder. As a result, decoder tries to reconstruct $x$ by ignoring useless $z$'s which are sampled from $\mathcal{N}(a,b)$.

Here are some explanations from Z-Forcing: Training Stochastic Recurrent Networks:

In these cases, the posterior approximation tends to provide a too weak or noisy signal, due to the variance induced by the stochastic gradient approximation. As a result, the decoder may learn to ignore z and instead to rely solely on the autoregressive properties of x, causing x and z to be independent, i.e. the KL term in Eq. 2 vanishes.


In various domains, such as text and images, it has been empirically observed that it is difficult to make use of latent variables when coupled with a strong autoregressive decoder.

where the simplest form of KL term, for the sake of clarity, is $$D_{KL}(q_{\phi}(z|x) \parallel p(z|x)) = D_{KL}(q_{\phi}(z|x) \parallel \mathcal{N}(0,1))$$ The paper uses a more complicated Gaussian prior for $p(z|x)$.

  • $\begingroup$ I think there are some mistakes in the answer by @Esmailian. The answer said $μ_d$ and $σ_d$ are stable functions, and then later that they are unstable, which is a contradiction. I think noisy signal means that $σ_d$ is too large. And the referenced paper seems to be wrong. The KL term that vanishes is not KL between learnt posterior and true posterior, it is KL between learnt posterior and the prior that is in Eq. 3 in the paper. $\endgroup$
    – calee
    Feb 21, 2022 at 1:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.