How to detect vanishing and exploding gradients with Tensorboard?

Question

I have two "sub-questions"

1) How can I detect vanishing or exploding gradients with Tensorboard, given the fact that currently write_grads=True is deprecated in the Tensorboard callback as per "un-deprecate write_grads for fit #31173" ?

2) I figured I can probably tell whether my model suffers from vanishing gradients based on the weights' distributions and histograms in the Distributions and Histograms tab in Tensorboard. My problem is that I have no frame of reference to compare with. Currently, my biases seem to be "moving" but I can't tell whether my kernel weights (Conv2D layers) are "moving"/"changing" "enough". Can someone help me by giving a rule of thumb to asses this visually in Tensorboard? I.e. if only the bottom 25% percentile of kernel weights are moving, that's good enough / not good enough? Or perhaps someone can post two reference images from tensorBoard of vanishing gradients vs, non vanishing gradients.

Here are my histograms and distributions, is it possible to tell whether my model suffers from Vanishing gradients? (some layers omitted for brevity) Thanks in advance.

Answer 1

As you are in search of exploding / vanishing gradients, it would be the best to check the gradient histogram, rather than the weights directly.

I found a code on Quora, pasting it just in case the link is gone

with tf.name_scope('train'): 
optimizer = tf.train.AdamOptimizer() 
# Get the gradient pairs (Tensor, Variable) 
grads = optimizer.compute_gradients(cross_entropy) 
# Update the weights wrt to the gradient 
train_step = optimizer.apply_gradients(grads) 
# Save the grads with tf.summary.histogram 
for index, grad in enumerate(grads): 
    tf.summary.histogram("{}-grad".format(grads[index][1].name), grads[index]) 

Just in case you would like to understand the vanishing / exploding gradients from the values, you can simply follow the logic as below:

• If you face with exploding gradients, it will increase some of the weights dramatically high, which will eventually hit to NaN and will make loss NaN. Hence, that would be a typical output of an exploding gradient.
• If you face with vanishing gradient, you shall observe that the weights of all or some of the layers to be completely same over few iteration / epoch. Please note that you cannot really set a rule as "%X percent to detect vanishing gradients", as the loss is based on the momentum and learning rate. Learning rate and/or momentum may be low enough to cause vanishing gradients for a while, then if any of them gets high enough, it may break the vanishing gradient problem. What you aim in a proper training is smoothly changing histograms over iterations, rather than constant weights and bias distributions over time.