I am trying to understand how to design mu and sigma vector in VAE autoencoder, as we know std value can not be larger than 0. If I use linear function, the value can be lower than zero, so I think Relu is the only activation function to apply in order to get sigma (Std) vector larger than zero. Would this be correct understanding?


1 Answer 1


Yeah, almost, except that Relu might cause problems ones it goes to zero (having zero variance is a little odd and the gradient goes to zero as well). Usually you'd go with

$$ \sigma = \exp(\rho) $$

where $\rho$ is coming from the encoder.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.