Where Does the Normal Glorot Initialization Come from?

Question

The famous Glorot initialization is described first in the paper Understanding the difficulty of training deep feedforward neural networks. In this paper, they derive the following uniform initialization, cf. Eq. (16) in their paper: \begin{equation} W \sim U\left[ -\frac{\sqrt{6}}{\sqrt{n_j + n_{j+1}}}, \frac{\sqrt{6}}{\sqrt{n_j + n_{j+1}}}\right]. \tag{16}\end{equation}

If we take a look at the PyTorch documentation for weight initialization, then there are two Glorot (Xavier) initializations, namely torch.nn.init.xavier_uniform_(tensor, gain=1.0) and torch.nn.init.xavier_normal_(tensor, gain=1.0). According to the documentation, the initialization for the latter is given by the normal distribution $\mathcal N(0, \sigma^2)$, where the standard deviation is given by $$ \sigma = \sqrt{\frac{2}{n_{j} + n_{j+1}}}.$$

Questions:

1.) Why do we have a $2$ instead of a $6$ in the standard deviation of the normal Glorot initialization?

2.) Where does the normal Glorot initialization come from? So basically, was there a follow-up paper to the above mentioned paper that demonstrated a superiority of the normal Glorot in comparison to the uniform initialization?

Thanks!

Answer 1

1.) Why do we have a 2 instead of a 6 in the standard deviation of the normal Glorot initialization?

I think you're just confused with notations: the way we parametrize a uniform distribution is using bounds and not moments. When we write $W \sim U(a,b)$, $a$ and $b$ are the lower and upper bounds of your distribution. The standard deviation of a uniform distribution is then $\sigma(W) = \frac{1}{\sqrt{12}}(b-a)$.

For the sake of readability let's define $c=n_j + n_{j+1}$, then for Glorot Uniform initialization: $$ W \sim U\left[ -\frac{\sqrt{6}}{\sqrt{c}}, \frac{\sqrt{6}}{\sqrt{c}}\right]$$ and we simply have: $$\sigma(W) = \frac{1}{\sqrt{12}}.\frac{2\sqrt{6}}{\sqrt{c}} = \sqrt{\frac{2}{c}}$$

Thus both normal and uniform Glorot initialization have the same variance!

2.) Where does the normal Glorot initialization come from? So basically, was there a follow-up paper to the above mentioned paper that demonstrated a superiority of the normal Glorot in comparison to the uniform initialization?

In fact in Glorot's paper, this variance/standard deviation is derived first without assumptions about the underlying distribution (see Eqn (12) in section 4.2.1 of Glorot's paper) and only later they propose to plug it in the uniform distribution in Eqn (16).

The book Deep Learning from Goodfellow et al, section 8.4 states:

We almost always initialize all the weights in the model to values drawn randomly from a Gaussian or uniform distribution. The choice of Gaussian or uniform distribution does not seem to matter very much, but has not been exhaustively studied. The scale of the initial distribution, however, does have a large effect on both the outcome of the optimization procedure and on the ability of the network to generalize.

So to summarize: there's no direct follow up paper on this, the normal version is just a normal distribution with the same variance, since the latter has been chosen for theoretical reasons, assuming the actual distribution doesn't really matter.

Then why Normal vs Uniform ? (personal opinion):

• Uniform: the point of random initialization is to break symmetry between several units of the same layer. Your best bet to make sure you break symmetry is to use the distribution with highest entropy, hence the uniform. Another argument is, that by having larger absolute weights the gradients will backpropagate better in a deep network.
• Normal: batch normalization effectiveness hints that having activation close to 0 is more efficient for learning (faster and less vanishing/exploding gradients problems), and with its exponential form the normal distribution will produce random weights (and thus activations) closer to 0