0
$\begingroup$

I am trying to understand how to design mu and sigma vector in VAE autoencoder, as we know std value can not be larger than 0. If I use linear function, the value can be lower than zero, so I think Relu is the only activation function to apply in order to get sigma (Std) vector larger than zero. Would this be correct understanding?

$\endgroup$
1
$\begingroup$

Yeah, almost, except that Relu might cause problems ones it goes to zero (having zero variance is a little odd and the gradient goes to zero as well). Usually you'd go with

$$ \sigma = \exp(\rho) $$

where $\rho$ is coming from the encoder.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.