I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & sigmoid.
But, I have encountered an explanation by Prof. Andrew NG lecture that explains that a partial solution to the vanishing gradient problem is a better or more careful choice of the random initialization of weights in your neural network.
i.e the solution is:
To set the variance of Wi to be equal to 1/n, where n is the number of input features that are going into a neuron. Along with the assumption that the input features of activations are roughly mean 0 and standard variance 1. So, what it's doing is that it's trying to set each of the weight matrices w so that it's not too much bigger than 1 and not too much less than 1, therefore, it doesn't explode or vanish too quickly.
- So, if you are using a ReLu activation function then setting the variance of Wi to be equal to
- and if you are using a TanH activation function then setting the variance of Wi to be equal to
- or in some cases, it's being suggested to use Xavier initialization
- Also, if we need we can tune of
variance parameteras another hyperparameter by multiplying into the above formula and tune that multiplier as part of your hyperparameter search.
Therefore, choosing a reasonable scaling for how to initialize the weights helps weights not to explode too quickly and not decay to zero too quickly, which in turn could help in training a reasonably deep network without the weights or the gradients exploding or vanishing too much and not simply using ReLu!. Please correct me if my understanding is wrong or incomplete!