In standard autoencoder, we encode data to bottleneck, then decode with using initial input as output to compute loss. We do activate matrix multiplication all over the network and if we are good, initial input should be close to output. I completely understand this.
But within a Vae, I have few problems, especially in understanding latent space vector. I think I know how to create it, but I am not really sure what is its purpose. Here I will write a few ideas, please let me know if I am correct.
We first create mu and sigma matrices, which are just matrix multiplication of previously hidden layer and random weights.
To create Z (latent matrix), we use parameterization trick. mu+log(0.5*sigma)*epsilon, which is a random matrix with 0 mean and 1 std. I have seen this Z (latent matrix) always produces Gaussian distribution, no matter what is the distribution of mu and sigma vectors. And here is my question. Why, why do we want to feed forward latent matrix with Gaussian distribution?
When we come to the end of decoding. We compute loss function and we penalize the network with KL divergence of Mu and Sigma Matrix. I don't understand why is it important in the first place to compare a distribution of Mu and Sigma matrices? I assume, that when we do back-propagation, mu and sigma become more close in term of distribution, but my question is why is this important, why those two matrixes must be close to each other in term of distribution?
I would really appreciate very simplistic answers with simple examples if it is possible.