I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.

The problem I observed depends on initial weights. When using sigmoid function it does not matter that much if weights are random numbers in ranges of [0,1] or [-1,1] or [-0.5,0.5]. But when using ReLu the network very often has a huge problem with ever converging when I'm using random weights in range [-1,1]. But when I changed the range of initialization of weights to [-0.5,0.5] it started to work. This only applies to ReLu activation function and I totally don't get it why it won't work for [-1,1]. Shouldn't it be able to converge with any random weights?

Also when I changed initial weights to normal distibution, it has no problem with convergence. I understand that normal distribution should work better and faster than random [-1,1]. What I don't understand is why it can't converge (error remains the same epoch after epoch) with [-1,1] and has no problem with converging with normal distribution... Shouldn't it always be able to converge just slower and faster with different initialization method?

PS. I'm using normal backpropagation with softmax as last layer and MSE as loss function

  • $\begingroup$ What is your learning rate? If the learning rate is too large, the NN will not converge. I suggest decreasing the learning rate twice and testing the results for random weights in [-1,1]. $\endgroup$ Dec 10, 2020 at 23:22
  • $\begingroup$ You say you use a softmax layer i.e a classification task, right? If yes, loss function has to be cross-entropy (negative log likelihood loss) $\endgroup$
    – sai
    Dec 11, 2020 at 0:33
  • $\begingroup$ @kate-melnykova Decrasing learning rate helps it converge and I understand the logic behind lowering learning rate. My case is that with the same learning rate but different initialization methods it works good or doesn't work at all. $\endgroup$ Dec 11, 2020 at 17:14
  • $\begingroup$ Could you please just try it and let me know the results? I have an idea of why it may be the case, but (a) it is lengthy to explain, (b) I am not 100% sure if it is the case. $\endgroup$ Dec 11, 2020 at 17:47
  • $\begingroup$ @kate-melnykova I tried it and it worked. For learning rate 0.01 and 0.001 it has a constrant error, so won't ever converge, when initialization is random in range [-1,1]. Only when I decrased learning rate 100x so lr=0.0001 it started to converge... For comparasion when using the same architecture but weights are initialized from normal distribution, learning rate lr=0.01 is perfectly good for NN to converge. Why does weight initialization matters so much not only in speed of convergence but even in success of ever converging? $\endgroup$ Dec 11, 2020 at 19:10

1 Answer 1


I will start with a toy example for the convergence part. Suppose that the loss function is $f(x) = x^4$ and we want to minimize it using the gradient descent. Clearly, the minimum is attained at zero and, in general, we would like the magnitude of the current approximation to decrease.

The update rule of gradient descent is $$ x_{k+1} = x_k - \lambda \nabla f = x_k - \lambda \cdot 4x_k^3.$$ Simplifying the expression, we get $$x^{k+1} = x^k(1 - 4\lambda x_k^2).$$ And now the combo initialization + learning rate start to appear. If $|1-4\lambda x_0^2| < 1$, then $|x_0| > |x_1| > |x_2| > ...$. The sequence will go to zero eventually. If $|1-4\lambda x_0^2| > 1$, then $|x_0| < |x_1|$. In this case, $|x_1| < |x_2|$ and so on -- the sequence will grow. Therefore, if the learning rate $\lambda$ if fixed, then the initial values $x^0$ determine if the grad descent converges or not.

When the gradient descent converge? The math says that the gradient descent converge when the learning rate $\lambda$ and the gradient $\nabla f$ satisfy $$\|\lambda \nabla f\| < 1$$ along the optimization path. Note that this condition does not need to be valid for all values of $x$, but at every $x_0, x_1, x_2, ...$

For many "good" functions, it suffices to require $\|\lambda \nabla f\| < 1$ only at $x_0$. The reason is that after the first iteration, we are closer to the local minimum, and for many "good" functions, it means that the gradient will be smaller.

What about weight choice in [-0.5,0.5] and [-1, 1]? I think of it as follows: Suppose that we selected weights in $[-0.5, 0.5]$ (model 1) and then multiplied all weights by 2 to get uniform distribution in $[-1, 1]$ (model 2). Suppose that the learning rate is identical in both cases, and let's check how the SGD performs. For simplicity of the argument, I replace it with the gradient descent.

How does it transfer to the NN? Note that a linear map (say, for the dense layer) has the following property. Suppose that all weights W are multiplied by 2, then $$\|2W\| = 2\|W\|.$$ ReLU is quazi-linearly-scalable: for every $a > 0$, $$a\cdot\text{ReLU}(x)=\text{ReLu}(ax).$$ Note that if your NN has $d$ layers, then, multiplying all weights by 2, your output increases in $2^d$ times (factor of 2 for each layer).

It is hard to compute the gradient for the NN, but using the product rule, I expect it to increase in approximately $2^{d}$ times. If model 1 satisfies the conditions for the gradient descent convergence, then we need to decrease the learning rate about $2^{d}$ times to guarantee the gradient descent convergence condition.

  • $\begingroup$ Thanks, that would explain why I have this problem! Now when I know this it's kind of obvious, but unfortunately I didn't think of that. $\endgroup$ Dec 11, 2020 at 20:03
  • $\begingroup$ It's me who should thank you. It is a nice question to think of, and as a math major, it is nice to connect theory with practice. $\endgroup$ Dec 11, 2020 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.