HOW TO: Deep Neural Network weight initialization

Question

Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train. To ease many of the problems one might:

1. Normalize && handpick quality data
2. choose a different training algorithm (e.g RMSprop instead of Gradient Descent)
3. pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)
4. Use different network structure (e.g Convolution layers instead of Feedforward)

I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better: Glorot and Bengio (2010)

• for sigmoid units: sample a Uniform(-r, r) with $r = \sqrt{\frac{6}{N_{in} + N_{out}}}$
• or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4 \sqrt{\frac{6}{N_{in} + N_{out}}}$

Is there any consistent way of initializing the weights better?

Answer 1

As far as I know the two formulas you gave are pretty much the standard initialization. I had done a literature review a while ago, I copied it below if interested.

---

[1] addresses the question:

First, weights shouldn't be set to zeros in order to break the symmetry when backprogragating:

Biases can generally be initialized to zero but weights need to be initialized carefully to break the symmetry between hidden units of the same layer. Because different output units receive different gradient signals, this symmetry breaking issue does not concern the output weights (into the output units), which can therefore also be set to zero.

Some initialization strategies:

• [2] and [3] recommend scaling by the inverse of the square root of the fan-in
• Glorot and Bengio (2010) and the Deep Learning Tutorials use a combination of the fan-in and fan-out:
  • for sigmoid units: sample a Uniform(-r, r) with $r=\sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
  • for hyperbolic tangent units: sample a Uniform(-r, r) with $r=4 \sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
• in the case of RBMs, a zero-mean Gaussian with a small standard deviation around 0.1 or 0.01 works well (Hinton, 2010) to initialize the weights.
• Orthogonal random matrix initialization, i.e. W = np.random.randn(ndim, ndim); u, s, v = np.linalg.svd(W) then use u as your initialization matrix.

Also, unsupervised pre-training may help in some situations:

An important choice is whether one should use unsupervised pre-training (and which unsupervised feature learning algorithm to use) in order to initialize parameters. In most settings we have found unsupervised pre-training to help and very rarely to hurt, but of course that implies additional training time and additional hyper-parameters.

Some ANN libraries also have some interesting lists, e.g. Lasagne:

Constant([val]) Initialize weights with constant value.
Normal([std, mean]) Sample initial weights from the Gaussian distribution.
Uniform([range, std, mean]) Sample initial weights from the uniform distribution.
Glorot(initializer[, gain, c01b])   Glorot weight initialization.
GlorotNormal([gain, c01b])  Glorot with weights sampled from the Normal distribution.
GlorotUniform([gain, c01b]) Glorot with weights sampled from the Uniform distribution.
He(initializer[, gain, c01b])   He weight initialization.
HeNormal([gain, c01b])  He initializer with weights sampled from the Normal distribution.
HeUniform([gain, c01b]) He initializer with weights sampled from the Uniform distribution.
Orthogonal([gain])  Intialize weights as Orthogonal matrix.
Sparse([sparsity, std]) Initialize weights as sparse matrix.

---

[1] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

[2] LeCun, Y., Bottou, L., Orr, G. B., and Muller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade.

[3] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." International conference on artificial intelligence and statistics. 2010.