# How to use Variational Autoencoder's μ and σ with user-generated z?

My understanding of VAE is that unlike Autoencoders, it does not directly give you a discrete encoding (latent code vectors n-dim) instead, it gives you both mu and sigma (n- dim mean vectors and n-dim standard deviation vectors). Then you have epsilon which you use to sample from a normal distribution with mu and sigma to create z. When combining mu, sigma and epsilon, you get z which is the one decoded by the VAE's decoder. z is basically the main encoding.

Say my z, mu, sigma are of n-dimension like 10 (10-dim z, mu, sigma). I enable the user to have a free picking/giving me numbers 10 vectors [-a, a], say a = 5. So the user is free to pick 10 vectors between -5, 5.

This becomes my z that is decoded by my decoder to generate a new image.

[Main problem]

My VAE is trained on a dataset of apparel. Now, if I run my VAE's encoder on each of the data in the dataset, I'd get a mu and sigma for each (not sure if this is still correct).

Using the z given by the user, how do I find the most similar from the dataset using VAE's encoding of only mu and sigma?

My thinking is to generate z using mu and sigma generated by VAE'S encoder but in order to generate z, I still need to sample from a distribution using epsilon in which makes it non-discrete w.r.t user-generated z. This adds randomness to it so I am not sure how would I use the encoded z to find the nearest to user-generated z.

• Not sure if I understand the scenario. So the user picks a random vector in the latent space (z-space) and you want to find the data point x which most likely generated this random vector? And x is from a given dataset? Dec 3, 2019 at 10:27
• Not exactly. The user gives the z-vector. I run my encoder to every data in my dataset to create mu-vector and sigma-vector. I need to know how to find the nearest encoded datapoint to the z-vector given by the user. Dec 3, 2019 at 17:03
• What do you mean by "near" though? Since the mu and sigma parameterize the posterior of z given x, likelihood would be a possible measure for distance between a given z and the encoding of x. Dec 3, 2019 at 17:22
• Most similar among the dataset, I mean. I was thinking of a distance metric. Dec 3, 2019 at 22:13

If you are training a VAE the encoder is essentially parameterizing the variational posterior distribution of $$z$$ given $$x$$, i.e. $$q(z | x) = \prod_{i=1}^{N_z} q(z_i | x) = \prod_{i=1}^{N_z} \frac{1}{\sqrt{2\pi}\sigma_i(x)} \exp \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} \right]$$ where $$\mu_i(x)$$ and $$\sigma_i(x)$$ are given by the encoder and $$z$$ is in the $$N_z$$-dimensional latent space. I would think of the problem as if the $$x$$ are parameters of a probability distribution and the $$z$$ are some observations you made. The "nearest" encoding from your training data $$x$$ would be the encoding with the highest likelihood, i.e. you compute the likelihood for each data point for a given $$z$$ by evaluating the above expression and take the $$x$$ with the maxmal value.

The log-likelihood is usually used in these scenarios because it's more convenient, but it is equivalent, as the likelihood is non-negative and the log is a monotonic function.

In the comments you mentioned to use a distance metric. The log-likelihood provides a nice interpretation, because it gives you something similar to the negative euclidean distance between $$\mu(x)$$ and $$z$$, but scales and shifts by terms determined by the standard deviation:

$$\log q(z|x) = \sum_{i=1}^{N_z} \log q(z_i|x) = \sum_{i=1}^{N_z} \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} - \log \left( \sqrt{2\pi}\sigma_i(x) \right) \right]$$

So intutively, by maximizing the (log-)likelihood, you are minimizing the euclidean distance between a given z and the encoding $$\mu(x)$$ of $$x$$ from the training data set, but you pay penalties for having large variances.

(Furthermore, if you do it this way, there is no sampling of $$\epsilon$$ required.)

• so should I just store each μi(x) and σi(x) for each dataset I have that I will encode after the training phase? With this, my questions are now... [1] So to find the nearest possible match among all encoded x (and its corresponding μi(x) and σi(x) vs the user-generated z, I would just compute logq(z|x) and get the smallest value? [2]. I would like to cluster my encodings using the encoder, but how do I do this? Should I just use μi(x) and σi(x) as features for x? Dec 4, 2019 at 3:31
• my autoencoder produces mean and log of variance of Q(z|X) with the following code def sampling(args): z_mean, z_log_var = args batch = K.shape(z_mean)[0] dim = K.int_shape(z_mean)[1] # by default, random_normal has mean = 0 and std = 1.0 epsilon = K.random_normal(shape=(batch, dim)) return z_mean + K.exp(0.5 * z_log_var) * epsilon Is z_log_var the sigma here? Dec 4, 2019 at 5:57
• To the first comment's first question: yes, BUT you have to take the maximal value. To the second qurstion: for that you should maybe open another question, but in short I would look at the KL-divergence between Gaussians. Dec 4, 2019 at 9:20
• To the second comment: that code does not generate the $\mu$ and $\sigma$. It instead takes $\mu$ and $\sigma$ as arguments and generates samples from the corresponding Gaussian distribution. z_log_var is the logarithm of $\sigma^2$, so you take the exp of that and get the variance. Dec 4, 2019 at 9:25
• if I have my VAE outputting z_log_sigma instead, how do I use my z_log_sigma to get the σ(x) needed for your proposed equation? Also, when you say maximal value, does this mean that the most similar is the one with the largest logq(z|x)? Dec 5, 2019 at 1:16