6
$\begingroup$

I'm new to datascience so please just don't blast me.

In a text book i found:

Now, the reason we don't initialize the weights to zero is that the learning rate (eta) only has an effect on the classification outcome if the weights are initialized to non-zero values. If all the weights are initialized to zero, the learning rate parameter eta affects only the scale of the weight vector, not the direction.

Now, why? Are you able to explain me?

$\endgroup$
3
$\begingroup$

If you initialize all weights with zeros then every hidden unit will get zero independent of the input. So, when all the hidden neurons start with the zero weights, then all of them will follow the same gradient and for this reason "it affects only the scale of the weight vector, not the direction".

Also, having zero ( or equal) weights to start with will prevent the network from learning. The errors backpropagated through the network is proportional to the value of the weights. If all the weights are the same, then the backpropagated errors will be the same, and consequently, all of the weights will be updated by the same amount. To avoid this symmetry problem, the initial weights to the network should be unequal.

Look at these links in more detail:

1) https://www.quora.com/Why-does-it-work-to-initialize-weights-of-a-deep-Neural-Network-to-zero-plus-some-noise-N-0-epsilon-and-not-anything-else

2) http://staff.itee.uq.edu.au/janetw/cmc/chapters/BackProp/index2.html

3) https://stackoverflow.com/questions/20027598/why-should-weights-of-neural-networks-be-initialized-to-random-numbers

4) https://stats.stackexchange.com/questions/27112/danger-of-setting-all-initial-weights-to-zero-in-backpropagation

$\endgroup$
  • $\begingroup$ great answer, but shouldn't [Something that does not depend on η] be [Something that does not depend on W(i)] and the direction is only determined by the inputs and targets? $\endgroup$ – pimajor Dec 10 '18 at 14:21
5
$\begingroup$

If you choose zero initial weights, then the perceptron algorithm's learning rate $\eta$ has no influence on a neuron's predicted class label.

This has to do with the fact that the decision function used in the perceptron algorithm, $$ \phi(z) = \begin{cases} 1 &\text{ if } z \ge 0\\ -1 &\text{ otherwise,} \end{cases} $$ depends only on the sign of $z$.

Consider a perceptron algorithm with a single neuron. (I will write ${\bf x} \cdot {\bf y}$ for the vector product ${\bf x}^T {\bf y}$ to avoid ugly double superscripts).

  • We pick an initial weight vector ${\bf w}^{(0)}.$
  • We feed in our first input vector ${\bf x^{(1)}}$ and predict the class label $$ \hat{y}^{(1)} = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}). $$ This gives us the weight update $$ \Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}, $$ where $\eta \in (0, 1)$ is the learning rate and $y^{(1)}$ is the true class label. The new weights are ${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$
  • Similarly, we feed in our second input vector ${{\bf x}^{(2)}}$ and predict the class label $$ \hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}). $$ This gives us the weight update $$ \Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)}, $$ where $y^{(2)}$ is the true class label.

Notice that $\Delta{\bf w}^{(2)}$ implicitly depends on $\Delta{\bf w}^{(1)}$, which in turn implicitly depends on ${\bf w}^{(0)}$. So let us unravel these dependencies by plugging in: \begin{align*} \Delta {\bf w}^{(2)} &= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\ &= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\ &= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\ &= \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\ &= \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}. \end{align*}

Now suppose we have initialized with zero weights: ${\bf w}^{(0)} = {\bf 0}.$ Since ${\bf 0} \cdot {\bf x}^{(1)} = 0$ and $\phi(0) = 1,$ the last line in this calculation simplifies to $$ \Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} $$

Let us zoom in on the function $\phi$ and its argument: $$ \phi\Big( \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big). $$ Since $\eta > 0$, it does not change the sign of $(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$ But the sign is all that matters to the function $\phi$. So we can simply remove $\eta$ from the function argument without changing the result: $$ \Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}. $$

The same arguments hold for $\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$

Conclude that $${\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$ for every $i = 1, 2, \dots$ This is what your quote means by saying that the parameter $\eta$ "affects only the scale of the weight vector, not the direction."

For nonzero initial weights, this is different. Recall the last line of the long calculation above and zoom in again on the function $\phi$ and its argument: $$ \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big). $$ If ${\bf w}^{(0)}$ is nonzero, then $\eta$ may affect the sign of the function argument, and therefore the predicted class label.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.