1
$\begingroup$

"The same value in all the parameters makes all the neurons have the same effect on the input, which causes the gradient with respect to all the weights is the same and, therefore, the parameters always change in the same way."

Taken from my course.

$\endgroup$
1
  • $\begingroup$ If the weights of the network are initialized to the same value they will all change in the same way when doing the backpropagation (because the gradient is the same). As a result of this the model isn't able to learn, therefore it is important to initialize the weights to random values. $\endgroup$
    – Oxbowerce
    Commented Jul 4, 2022 at 16:37

2 Answers 2

0
$\begingroup$

Consider the following image of a simple neural network. Note that the network uses a linear activation function and that there are no bias terms (this makes the intuition easier).

enter image description here

Each path from the input to the output is as follows $$ f(x) = (x*w1)*w4 = (x*0.5)*0.5 $$ $$ f(x) = (x*w2)*w5 = (x*0.5)*0.5 $$ $$ f(x) = (x*w3)*w6 = (x*0.5)*0.5 $$

When you perform gradient descent, the change in the weights will always be the same as each path is identical.

$\endgroup$
3
  • $\begingroup$ Is there a source for the figure, or is it yours? Thanks! $\endgroup$
    – hH1sG0n3
    Commented Jul 4, 2022 at 18:06
  • $\begingroup$ But that is irrelevant for multi-input, isn't it? $\endgroup$ Commented Jul 4, 2022 at 18:10
  • $\begingroup$ I put the image together using Canva! And for multi-input the statement in the question would be false assuming that all the inputs are different. $\endgroup$
    – BoomBoxBoy
    Commented Jul 4, 2022 at 19:18
0
$\begingroup$

If all the weights contribute equally, there is no way to single out some specific weights to penalise during backpropagation. Therefore the change in weights is just a trivial global rescaling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.