So I understood that when training VAE, we need to weight the KL part of the loss with a weight less than 1 so that the reconstruction loss can get a chance to learn (avoiding the posterior collapse).
When I tried to do so in different techniques (constant beta, beta increasing from 0 to 1 (warm-up), I saw that the KL loss and the reconstruction loss are always in opposite directions, i.e, when the beta is small enough to let the reconstruction converge, the KL naturally increases, and when the beta increases the KL decreases and the recons increases back. For instance from a warm-up beta scenario:
Training reconstruction loss :
Training KL loss:
Total loss (recons + beta* kl):
And the beta value along the training epochs:
Intuitively I assumed the model is just bad as I can't get it to have both kl and reconstruction small. However I saw in here for instance: https://www.microsoft.com/en-us/research/blog/less-pain-more-gain-a-simple-method-for-vae-training-with-less-of-that-kl-vanishing-agony/ that their cycling beta method may cause the reconstruction and the ELBO to decrease but the KL keeps on increasing. I also saw in a different presentation that they only show the ELBO loss behavior throughout the training process as if the KL trend is irrelevant.
I guess my questions are :
- Does the behavior I'm showing make sense?
- Should I aim for having only low reconstruction and ELBO ? Or am I missing something interpreting other's results?
Thanks in advance for any help.
