Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am new in this field so please be gentle with terminology. In the original paper; "Understanding the difficulty of training deep feedforward neural networks", I dont understand how equation 15 is obtained, it states that giving eq 1 :
$$ W_{ij} \sim U\left[−\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\right] $$
it gives rise to variance with the following property:
$$ n*Var[W]=1/3 $$
where $n$ is the size of the layer.
How is this last equation(15) obtained?
Thanks!!
The fixed answer of $1/3$ is a result of their decision to use the uniform distribution along with the parameterised arguments, namely $1/\sqrt{n}$.
For a uniform distribution, denoted with lower and upper bounds $a$ and $b$:
$$ U(a, b) $$
the variance is defined as:
$$ \frac{1}{12} (b - a)^2 $$
So in the case of the authors, Glorot and Bengio, the two bounds are simply the square root of the number of neurons in the layer of interest (generally referring to the preceding layer, as they put it). This size is called $n$, and they set the bounds on the uniform distribution as:
$$ a = - \frac{1}{\sqrt{n}} $$ $$ b = \frac{1}{\sqrt{n}} $$
So if we plug these values into equation 15, we get:
$$ Var = \frac{1}{12}(\frac{1}{\sqrt{n}} - -\frac{1}{\sqrt{n}})^2 $$
$$ Var = \frac{1}{12}(\frac{2}{\sqrt{n}})^2 $$
$$ Var = \frac{1}{12} * \frac{4}{n} $$
And so finally:
$$ n * Var = \frac{1}{3} $$
