My understanding of VAE is that unlike Autoencoders, it does not directly give you a discrete encoding (latent code vectors n-dim) instead, it gives you both mu and sigma (n- dim mean vectors and n-dim standard deviation vectors). Then you have epsilon which you use to sample from a normal distribution with mu and sigma to create z. When combining mu, sigma and epsilon, you get z which is the one decoded by the VAE's decoder. z is basically the main encoding.

Say my z, mu, sigma are of n-dimension like 10 (10-dim z, mu, sigma). I enable the user to have a free picking/giving me numbers 10 vectors [-a, a], say a = 5. So the user is free to pick 10 vectors between -5, 5.

This becomes my z that is decoded by my decoder to generate a new image.

[Main problem]

My VAE is trained on a dataset of apparel. Now, if I run my VAE's encoder on each of the data in the dataset, I'd get a mu and sigma for each (not sure if this is still correct).

Using the z given by the user, how do I find the most similar from the dataset using VAE's encoding of only mu and sigma?

My thinking is to generate z using mu and sigma generated by VAE'S encoder but in order to generate z, I still need to sample from a distribution using epsilon in which makes it non-discrete w.r.t user-generated z. This adds randomness to it so I am not sure how would I use the encoded z to find the nearest to user-generated z.

  • $\begingroup$ Not sure if I understand the scenario. So the user picks a random vector in the latent space (z-space) and you want to find the data point x which most likely generated this random vector? And x is from a given dataset? $\endgroup$
    – matthiaw91
    Dec 3, 2019 at 10:27
  • $\begingroup$ Not exactly. The user gives the z-vector. I run my encoder to every data in my dataset to create mu-vector and sigma-vector. I need to know how to find the nearest encoded datapoint to the z-vector given by the user. $\endgroup$ Dec 3, 2019 at 17:03
  • $\begingroup$ What do you mean by "near" though? Since the mu and sigma parameterize the posterior of z given x, likelihood would be a possible measure for distance between a given z and the encoding of x. $\endgroup$
    – matthiaw91
    Dec 3, 2019 at 17:22
  • $\begingroup$ Most similar among the dataset, I mean. I was thinking of a distance metric. $\endgroup$ Dec 3, 2019 at 22:13

1 Answer 1


If you are training a VAE the encoder is essentially parameterizing the variational posterior distribution of $z$ given $x$, i.e. $$ q(z | x) = \prod_{i=1}^{N_z} q(z_i | x) = \prod_{i=1}^{N_z} \frac{1}{\sqrt{2\pi}\sigma_i(x)} \exp \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} \right] $$ where $\mu_i(x)$ and $\sigma_i(x)$ are given by the encoder and $z$ is in the $N_z$-dimensional latent space. I would think of the problem as if the $x$ are parameters of a probability distribution and the $z$ are some observations you made. The "nearest" encoding from your training data $x$ would be the encoding with the highest likelihood, i.e. you compute the likelihood for each data point for a given $z$ by evaluating the above expression and take the $x$ with the maxmal value.

The log-likelihood is usually used in these scenarios because it's more convenient, but it is equivalent, as the likelihood is non-negative and the log is a monotonic function.

In the comments you mentioned to use a distance metric. The log-likelihood provides a nice interpretation, because it gives you something similar to the negative euclidean distance between $\mu(x)$ and $z$, but scales and shifts by terms determined by the standard deviation:

$$ \log q(z|x) = \sum_{i=1}^{N_z} \log q(z_i|x) = \sum_{i=1}^{N_z} \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} - \log \left( \sqrt{2\pi}\sigma_i(x) \right) \right] $$

So intutively, by maximizing the (log-)likelihood, you are minimizing the euclidean distance between a given z and the encoding $\mu(x)$ of $x$ from the training data set, but you pay penalties for having large variances.

(Furthermore, if you do it this way, there is no sampling of $\epsilon$ required.)

  • $\begingroup$ so should I just store each μi(x) and σi(x) for each dataset I have that I will encode after the training phase? With this, my questions are now... [1] So to find the nearest possible match among all encoded x (and its corresponding μi(x) and σi(x) vs the user-generated z, I would just compute logq(z|x) and get the smallest value? [2]. I would like to cluster my encodings using the encoder, but how do I do this? Should I just use μi(x) and σi(x) as features for x? $\endgroup$ Dec 4, 2019 at 3:31
  • $\begingroup$ my autoencoder produces mean and log of variance of Q(z|X) with the following code def sampling(args): z_mean, z_log_var = args batch = K.shape(z_mean)[0] dim = K.int_shape(z_mean)[1] # by default, random_normal has mean = 0 and std = 1.0 epsilon = K.random_normal(shape=(batch, dim)) return z_mean + K.exp(0.5 * z_log_var) * epsilon Is z_log_var the sigma here? $\endgroup$ Dec 4, 2019 at 5:57
  • $\begingroup$ To the first comment's first question: yes, BUT you have to take the maximal value. To the second qurstion: for that you should maybe open another question, but in short I would look at the KL-divergence between Gaussians. $\endgroup$
    – matthiaw91
    Dec 4, 2019 at 9:20
  • $\begingroup$ To the second comment: that code does not generate the $\mu$ and $\sigma$. It instead takes $\mu$ and $\sigma$ as arguments and generates samples from the corresponding Gaussian distribution. z_log_var is the logarithm of $\sigma^2$, so you take the exp of that and get the variance. $\endgroup$
    – matthiaw91
    Dec 4, 2019 at 9:25
  • $\begingroup$ if I have my VAE outputting z_log_sigma instead, how do I use my z_log_sigma to get the σ(x) needed for your proposed equation? Also, when you say maximal value, does this mean that the most similar is the one with the largest logq(z|x)? $\endgroup$ Dec 5, 2019 at 1:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.