2
$\begingroup$

I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values? I am getting values close to 4 decimal places(0.0001) and in some cases close to 5 decimal places (0.00001). The network seems not to be learning since the histogram of weight is also quite similar in all epochs. I am using RELU activation and Adam optimizer. What could be the reason for the vanishing gradient in case of RELU activation? Is possible please point me to some resources that might be helpful. Thanks in advance.

enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ What are the axis meaning of the plots? $\endgroup$
    – Javier TG
    Oct 22, 2020 at 11:57
  • 1
    $\begingroup$ x axis is value and y axis is epoch $\endgroup$
    – pramesh
    Oct 23, 2020 at 3:44

2 Answers 2

Reset to default
1
$\begingroup$

Vanishing gradients are the updates to the weights (not the weight themselves) over the layers after each training batch.

You should check the update signal between the top-most layer and lowest layer after a single batch.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ since during backprop gradients are multiplied, isn't is correct to interpret the small gradient values as vanishing gradient. I've been checking the update ratio as well and it is in 1e-5 scale. My main question is the point at which it can be declared as vanishing gradient problem by looking at the values $\endgroup$
    – pramesh
    Oct 25, 2020 at 14:41
0
$\begingroup$

Vanishing gradient as explained in the Xavier Glorot paper "Understanding the difficulty of training deep feedforward neural networks" can be seen as a shift of the mean of the gradient to lower values with small standard deviations when considering each layers from the output to the input.

There is no specific threashold to measure the vanishing gradient since it depends on you data, loss, network, etc. Only monitoring the mean- and std- of the gradient on each layer during multiple training epochs can help to make you own interpretation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.