I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
The problem I observed depends on initial weights. When using sigmoid function it does not matter that much if weights are random numbers in ranges of [0,1] or [-1,1] or [-0.5,0.5]. But when using ReLu the network very often has a huge problem with ever converging when I'm using random weights in range [-1,1]. But when I changed the range of initialization of weights to [-0.5,0.5] it started to work. This only applies to ReLu activation function and I totally don't get it why it won't work for [-1,1]. Shouldn't it be able to converge with any random weights?
Also when I changed initial weights to normal distibution, it has no problem with convergence. I understand that normal distribution should work better and faster than random [-1,1]. What I don't understand is why it can't converge (error remains the same epoch after epoch) with [-1,1] and has no problem with converging with normal distribution... Shouldn't it always be able to converge just slower and faster with different initialization method?
PS. I'm using normal backpropagation with softmax as last layer and MSE as loss function