What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

Question

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & sigmoid.

But, I have encountered an explanation by Prof. Andrew NG lecture that explains that a partial solution to the vanishing gradient problem is a better or more careful choice of the random initialization of weights in your neural network.

i.e the solution is:

To set the variance of Wi to be equal to 1/n, where n is the number of input features that are going into a neuron. Along with the assumption that the input features of activations are roughly mean 0 and standard variance 1. So, what it's doing is that it's trying to set each of the weight matrices w so that it's not too much bigger than 1 and not too much less than 1, therefore, it doesn't explode or vanish too quickly.

• So, if you are using a ReLu activation function then setting the variance of Wi to be equal to sqrt(2/n) works better**.
• and if you are using a TanH activation function then setting the variance of Wi to be equal to sqrt(2/n) works better.
• or in some cases, it's being suggested to use Xavier initialization
• Also, if we need we can tune of variance parameter as another hyperparameter by multiplying into the above formula and tune that multiplier as part of your hyperparameter search.

Therefore, choosing a reasonable scaling for how to initialize the weights helps weights not to explode too quickly and not decay to zero too quickly, which in turn could help in training a reasonably deep network without the weights or the gradients exploding or vanishing too much and not simply using ReLu!. Please correct me if my understanding is wrong or incomplete!

Answer 1

I think the two aspects you mention are two faces of the same medal: if your weights are too high or low, the activation of a layer ends up being too high or low. If you use $tanh(z)$ or $sigmoid(z)$ as activation functions, you'll end up with values that will have a derivative almost to zero, and the gradient will vanish as it is back-propagated into the network. By using $relu(z)$, instead, you use a linear function for $z>0$ and therefore the derivative is constant as the activation grows. This helps the gradient which does not approach zero while it is back-propagated.

This works in every case: both if you use weights that are too high or too small. Tanh and sigmoid are similar in the sense that they are basically linear near $z=0$, but almost flat when $|z|>>0$, ReLU solves the problem in the latter case, because it becomes linear for any $z>0$.

On the other hand, another strategy is to prevent $z$ from being too large or too small, and therefore you attempt to keep it near the origin ($z=0$). In this way, both tanh and sigmoid are linear, as ReLU would have been for any $z>0$.

The expected result is the same: the derivative of the function is almost linear, and therefore the gradients can traverse the network without vanishing or exploding.

Of course, there are other factors here that must be considered (e.g. ReLU clamps to 0 every negative value, which has other sort of effects), but beside these, in both cases you get a similar outcome (non-vanishing gradients) with different approaches (by tweaking the activation function or by tweaking the weights).