1
$\begingroup$

In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -

To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are defined to be copies of W but with each $W^{(t)}$ used only at time step t. We may then use $∇_{W^{(t)}}$ to denote the contribution of the weights at time step t to the gradient

What is the purpose of defining such a dummy variable?

Moreover, it seems like in the equation (10.26) in the book,

$∇_{W^{(t)}}h(t)$ = $diag(1-(h^{(t)})^2)h^{(t−1)^{T}}$.

Shouldn't it be $∇_{W^{(t)}}h^{(t)}$ = $diag(1-(h^{(t)})^2)(h^{(t−1)^{T}}+ W ∇_{W^{(t)}}h^{(t−1)})$ instead? Some backpropagation derivations, infact, apply chain rule in this manner (e.g. https://arxiv.org/pdf/1610.02583.pdf). What is the difference between such derivations with the derivation depicted in the deep learning book?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.