I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values? I am getting values close to 4 decimal places(0.0001)
and in some cases close to 5 decimal places (0.00001)
. The network seems not to be learning since the histogram of weight is also quite similar in all epochs. I am using RELU
activation and Adam
optimizer. What could be the reason for the vanishing gradient in case of RELU
activation? Is possible please point me to some resources that might be helpful. Thanks in advance.
-
$\begingroup$ What are the axis meaning of the plots? $\endgroup$– Javier TGOct 22, 2020 at 11:57
-
1$\begingroup$ x axis is value and y axis is epoch $\endgroup$– prameshOct 23, 2020 at 3:44
2 Answers
Vanishing gradients are the updates to the weights (not the weight themselves) over the layers after each training batch.
You should check the update signal between the top-most layer and lowest layer after a single batch.
-
1$\begingroup$ since during backprop gradients are multiplied, isn't is correct to interpret the small gradient values as vanishing gradient. I've been checking the update ratio as well and it is in 1e-5 scale. My main question is the point at which it can be declared as vanishing gradient problem by looking at the values $\endgroup$– prameshOct 25, 2020 at 14:41
Vanishing gradient as explained in the Xavier Glorot paper "Understanding the difficulty of training deep feedforward neural networks" can be seen as a shift of the mean of the gradient to lower values with small standard deviations when considering each layers from the output to the input.
There is no specific threashold to measure the vanishing gradient since it depends on you data, loss, network, etc. Only monitoring the mean- and std- of the gradient on each layer during multiple training epochs can help to make you own interpretation.