In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -

To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to be copies of W but with each $W^{(t)}$ used only at time step t. We may then use $∇_{W^{(t)}}$ to denote the contribution of the weights at time step t to the gradient

What is the purpose of defining such a dummy variable?

Moreover, it seems like in the equation (10.26) in the book,

$∇_{W^{(t)}}h(t)$ = $diag(1-(h^{(t)})^2)h^{(t−1)^{T}}$.

Shouldn't it be $∇_{W^{(t)}}h^{(t)}$ = $diag(1-(h^{(t)})^2)(h^{(t−1)^{T}}+ W ∇_{W^{(t)}}h^{(t−1)})$ instead? Some backpropagation derivations, infact, apply chain rule in this manner (e.g. https://arxiv.org/pdf/1610.02583.pdf). What is the difference between such derivations with the derivation depicted in the deep learning book?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.