I'm now learning about deep learning with Keras, and to implement a deep learning model at Keras, you set the initializer to set its initial weights on.

from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer="random_uniform"))

The kernel_initializer can take something others, such as random_normal, which uses Gaussian, not uniform distribution, and zero, which literally sets all weights to 0.

However, I don't understand why you like to set different weights at the initializer. Specifically, what advantages does it have over setting all initial weights to 0, which sounds more natural for novices like me?

Also, should the initial weights, if needed, be always set a tiny value (e.g. 0.05) to?

up vote 7 down vote accepted

This is greatly addressed in the Stanford CS class CS231n:

Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

There are several weight initialization strategies; each one is best suited for a type of activation function. For instance, Glorot's initialization aims at not saturating sigmoid activations, while He's initialization is meant for Rectified Linear Units (ReLUs).

That has to do with how forward and backpropagation works. Remember that forward propagation is done by applying the activation function to the result of multiplying the activations of layer al by a weight matrix Wl plus the bias vector bl for each layer of the network:

zl = activation(Wl * al-1 + bl)

where zl is the output of layer l and a0 would be the input layer. It is easy to see from there that if the Wl is set to zero the output zl will depend only on bl and the activation values will be the same for every neuron in the network.

The backpropagation step of gradient descent uses zl calculate the gradient step and parameter updates so if every neuron in the layer has the same value those steps will also output equal values and all the neurons will learn the same parameters.

By initializing the outputs to random values in the range you'll make each neuron learn differently breaking the symmetry.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.