Vanishing gradients: examine output gradients

Question

For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it is appropriate to feed the input 'xtr' into the backend function defined on weights.

weights_vars= model.layers[1].trainable_weights  # weights on 2nd hidden layer
sess= k.get_session()
# Obtain the actual gradients:
grad_fun= k.gradients(model.output, weights_vars[0])  # [0] for weight, [1] for bias
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})

I have seen posts demonstrating how to obtain gradients of output wrt $\textit{inputs}$, aka Jacobians. Feeding inputs to function defined on model.input is certainly correct.

grad_fun= k.gradients(model.output, model.input)
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})

My questions are:

• Can I use these Jacobians to check the extent of vanishing gradients, as a proxy to the gradients with respect to weights?
• How can I use backend.function defined on weights to obtain gradients? What do I put in feed_dict? If there is a better way to examine the output gradients on weights please let me know. Thanks in advance.

Answer 1

Jacobians are proxy to gradients. Sometimes in NN, we need to find partial derivatives of a function whose input and output both are vectors. The matrix containing all such partial derivatives are known as Jacobian Matrix. So yes you can use jacobian matrix as a proxy to graident.

The second question has already being answer here