# What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & sigmoid.

But, I have encountered an explanation by Prof. Andrew NG lecture that explains that a partial solution to the vanishing gradient problem is a better or more careful choice of the random initialization of weights in your neural network.

i.e the solution is:

To set the variance of Wi to be equal to 1/n, where n is the number of input features that are going into a neuron. Along with the assumption that the input features of activations are roughly mean 0 and standard variance 1. So, what it's doing is that it's trying to set each of the weight matrices w so that it's not too much bigger than 1 and not too much less than 1, therefore, it doesn't explode or vanish too quickly.

• So, if you are using a ReLu activation function then setting the variance of Wi to be equal to sqrt(2/n) works better**.
• and if you are using a TanH activation function then setting the variance of Wi to be equal to sqrt(2/n) works better.
• or in some cases, it's being suggested to use Xavier initialization
• Also, if we need we can tune of variance parameter as another hyperparameter by multiplying into the above formula and tune that multiplier as part of your hyperparameter search.

Therefore, choosing a reasonable scaling for how to initialize the weights helps weights not to explode too quickly and not decay to zero too quickly, which in turn could help in training a reasonably deep network without the weights or the gradients exploding or vanishing too much and not simply using ReLu!. Please correct me if my understanding is wrong or incomplete!

I think the two aspects you mention are two faces of the same medal: if your weights are too high or low, the activation of a layer ends up being too high or low. If you use $$tanh(z)$$ or $$sigmoid(z)$$ as activation functions, you'll end up with values that will have a derivative almost to zero, and the gradient will vanish as it is back-propagated into the network. By using $$relu(z)$$, instead, you use a linear function for $$z>0$$ and therefore the derivative is constant as the activation grows. This helps the gradient which does not approach zero while it is back-propagated.
This works in every case: both if you use weights that are too high or too small. Tanh and sigmoid are similar in the sense that they are basically linear near $$z=0$$, but almost flat when $$|z|>>0$$, ReLU solves the problem in the latter case, because it becomes linear for any $$z>0$$.
On the other hand, another strategy is to prevent $$z$$ from being too large or too small, and therefore you attempt to keep it near the origin ($$z=0$$). In this way, both tanh and sigmoid are linear, as ReLU would have been for any $$z>0$$.