If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables.

What I do not understand it, how you can calculate KL divergence in VAE (latent space vector and N(0,1) as it is stated in many tutorials.

Latent space vector is not a vector of random variables. It is a vector of product of input, weights,bias and activation functions. All these do not make your vector a random variable vector. My question is, how to properly create latent space vector as random variable vector, so you could eventually calculate KL divergence.


1 Answer 1


You are right that the output of your encoder neural network is not a random variable, this is the mean $\mu$ and the standard deviation $\sigma$ of your latent random variable. For example if the output of your encoder is $\sigma = 1$ and $\mu = 0.5$, your latent random variable will be normal with mean 0.5 and standard deviation 1. Theb, you can calculate the KL divergence between your random variable and the prior for the latent random variable which is $\mathcal{N}(0, 1)$ in many tutorials.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.