I am trying to understand how to design mu and sigma vector in VAE autoencoder, as we know std value can not be larger than 0. If I use linear function, the value can be lower than zero, so I think Relu is the only activation function to apply in order to get sigma (Std) vector larger than zero. Would this be correct understanding?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yeah, almost, except that Relu might cause problems ones it goes to zero (having zero variance is a little odd and the gradient goes to zero as well). Usually you'd go with
$$ \sigma = \exp(\rho) $$
where $\rho$ is coming from the encoder.
