If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can weights take on any value? Could a network potentially have weights of 0.1, 0.0009, and 100000?
One of the problems that can occur when training a neural network is known as the exploding gradient problem. A poorly initialised network could lead to a large increase in the norm of the gradient during training. These larger values will basically run the weights out of the number precision of the computer, resulting in NaN values.