For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it is appropriate to feed the input 'xtr' into the backend function defined on weights.
weights_vars= model.layers[1].trainable_weights # weights on 2nd hidden layer
sess= k.get_session()
# Obtain the actual gradients:
grad_fun= k.gradients(model.output, weights_vars[0]) # [0] for weight, [1] for bias
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})
I have seen posts demonstrating how to obtain gradients of output wrt $\textit{inputs}$, aka Jacobians. Feeding inputs to function defined on model.input is certainly correct.
grad_fun= k.gradients(model.output, model.input)
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})
My questions are:
- Can I use these Jacobians to check the extent of vanishing gradients, as a proxy to the gradients with respect to weights?
- How can I use backend.function defined on weights to obtain gradients? What do I put in
feed_dict
? If there is a better way to examine the output gradients on weights please let me know. Thanks in advance.