0
$\begingroup$

For a feedforward network or RNN, in theory we should examine the output gradients with respect to the weights over time to check whether it vanishes to zero. In my code below I am not sure whether it is appropriate to feed the input 'xtr' into the backend function defined on weights.

weights_vars= model.layers[1].trainable_weights  # weights on 2nd hidden layer
sess= k.get_session()
# Obtain the actual gradients:
grad_fun= k.gradients(model.output, weights_vars[0])  # [0] for weight, [1] for bias
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})

I have seen posts demonstrating how to obtain gradients of output wrt $\textit{inputs}$, aka Jacobians. Feeding inputs to function defined on model.input is certainly correct.

grad_fun= k.gradients(model.output, model.input)
grad_value= sess.run(grad_fun, feed_dict={model.input: xtr})

My questions are:

  • Can I use these Jacobians to check the extent of vanishing gradients, as a proxy to the gradients with respect to weights?
  • How can I use backend.function defined on weights to obtain gradients? What do I put in feed_dict? If there is a better way to examine the output gradients on weights please let me know. Thanks in advance.
$\endgroup$

1 Answer 1

1
$\begingroup$

Jacobians are proxy to gradients. Sometimes in NN, we need to find partial derivatives of a function whose input and output both are vectors. The matrix containing all such partial derivatives are known as Jacobian Matrix. So yes you can use jacobian matrix as a proxy to graident.

The second question has already being answer here

$\endgroup$
1
  • $\begingroup$ Thanks for the clarification. And it looks like in the first block my code to obtain the output gradients wrt to weights is correct. $\endgroup$
    – siegfried
    Jan 30, 2022 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.