3
$\begingroup$

If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can weights take on any value? Could a network potentially have weights of 0.1, 0.0009, and 100000?

$\endgroup$
1
  • $\begingroup$ Neural networks generally use random initialization for the weights. The weights take on values so that the sum of the weights times function values is 1. There is some good reading out there on how it all works, but it's basically just a probability model. $\endgroup$
    – Jabernet
    Feb 9 '19 at 23:25
2
$\begingroup$

One of the problems that can occur when training a neural network is known as the exploding gradient problem. A poorly initialised network could lead to a large increase in the norm of the gradient during training. These larger values will basically run the weights out of the number precision of the computer, resulting in NaN values.

This post gives more information on the exploding gradient problem and how to solve it. A related post discusses different initialization strategies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.