Initialize perceptron weights with zero

Question

I'm new to datascience so please just don't blast me.

In a text book i found:

Now, the reason we don't initialize the weights to zero is that the learning rate (eta) only has an effect on the classification outcome if the weights are initialized to non-zero values. If all the weights are initialized to zero, the learning rate parameter eta affects only the scale of the weight vector, not the direction.

Now, why? Are you able to explain me?

Answer 1

If you initialize all weights with zeros then every hidden unit will get zero independent of the input. So, when all the hidden neurons start with the zero weights, then all of them will follow the same gradient and for this reason "it affects only the scale of the weight vector, not the direction".

Also, having zero ( or equal) weights to start with will prevent the network from learning. The errors backpropagated through the network is proportional to the value of the weights. If all the weights are the same, then the backpropagated errors will be the same, and consequently, all of the weights will be updated by the same amount. To avoid this symmetry problem, the initial weights to the network should be unequal.

Look at these links in more detail:

1) https://www.quora.com/Why-does-it-work-to-initialize-weights-of-a-deep-Neural-Network-to-zero-plus-some-noise-N-0-epsilon-and-not-anything-else

2) http://staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html

3) https://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers

4) https://stats.stackexchange.com/questions/27112/danger-of-setting-all-initial-weights-to-zero-in-backpropagation

Answer 2

If you choose zero initial weights, then the perceptron algorithm's learning rate $\eta$ has no influence on a neuron's predicted class label.

This is because the decision function used in the perceptron algorithm depends only on the sign of $z$: $$ \phi(z) = \begin{cases} 1 &\text{ if } z \ge 0\\ -1 &\text{ otherwise.} \end{cases} $$

Consider a perceptron algorithm with a single neuron. (I will write ${\bf x} \cdot {\bf y}$ for the vector product ${\bf x}^T {\bf y}$ to avoid ugly double superscripts).

• Pick an initial weight vector ${\bf w}^{(0)}.$
• Plug in the first input vector ${\bf x^{(1)}}$ and predict the class label $$ \hat{y}^{(1)} = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}). $$ This gives the weight update $$ \Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}, $$ where $\eta \in (0, 1)$ is the learning rate and $y^{(1)}$ is the true class label. The new weights are ${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$
• Similarly, plug in our the input vector ${{\bf x}^{(2)}}$ and predict the class label $$ \hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}). $$ This gives the weight update $$ \Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)}, $$ where $y^{(2)}$ is the true class label.

Notice that $\Delta{\bf w}^{(2)}$ implicitly depends on $\Delta{\bf w}^{(1)}$, which in turn implicitly depends on ${\bf w}^{(0)}$. Let us unravel these dependencies by plugging in: \begin{align*} \Delta {\bf w}^{(2)} &= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\ &= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\ &= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\ &= \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\ &= \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}. \end{align*}

Now suppose we have initialized with zero weights: ${\bf w}^{(0)} = {\bf 0}.$ Since ${\bf 0} \cdot {\bf x}^{(1)} = 0$ and $\phi(0) = 1,$ the last line in this calculation simplifies to $$ \Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} $$

Let us zoom in here and look at $\phi$ and its argument: $$ \phi\Big( \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big). $$ Since $\eta > 0$, it does not change the sign of $(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$ But the sign is all that matters to the function $\phi$. So we can simply remove $\eta$ from the function argument without changing the result: $$ \Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}. $$

The same arguments hold for $\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$

It follows that $$\Delta {\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$ for every $i = 1, 2, \dots$ This is what your quote means by saying that the parameter $\eta$ "affects only the scale of the weight vector, not the direction."

For nonzero initial weights, this is different. Recall the last line of the long calculation $\Delta {\bf w}^{(2)} = \dots$ above and zoom in again on the function $\phi$ and its argument: $$ \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big). $$ If ${\bf w}^{(0)}$ is nonzero, then $\eta$ may affect the sign of the function argument, and therefore the predicted class label.

Answer 3

https://www.coursera.org/learn/neural-networks-deep-learning/lecture/XtFPI/random-initialization.

all the neurones will have same values in a hidden layer( after single/ all iteration, as derivate will be same). As both the neurones learning same thing, There is no contribution towards gradient but only on scale.

Its like having only one neuron in every layer.