# Why should the initialization of weights and bias be chosen around 0?

To train our neural network, we will initialize each parameter W(l)ijWij(l) and each b(l)ibi(l) to a small random value near zero (say according to a Normal(0,ϵ2)Normal(0,ϵ2) distribution for some small ϵϵ, say 0.01)

from Stanford Deep learning tutorials at the 7th paragraph in the Backpropagation Algorithm

What I don't understand is why the initialization of the weight or bias should be around 0?

Another potential issue is that the distribution of the outputs of each neuron, when using random initialization values, has a variance that gets larger with more inputs. A common additional step is to normalize the neuron's output variance to 1 by dividing its weights by $sqrt(d)$ where $d$ is the number of inputs to the neuron. The resulting weights are normally distributed between $\left[\frac{-1}{\sqrt{d}}, \frac{1}{\sqrt{d}}\right]$