0
$\begingroup$

I'm trying to implement RNN and LSTM , many-to-many architecture. I reasoned myself why BPTT is necessary in RNNs and it makes sense.

But what doesn't make sense to me is, most of resources I went through online for LSTM back prop (like attachment ) and this one : https://www.youtube.com/watch?v=8rQPJnyGLlY

Seems to be doing more of back propagation w.r.t current time stamp itself, but not through time , keeping variable convention aside, similar to RNN , I would assume that when calculating DL w.r.t Wf, since both current time step and previous hidden states are made up of Wf, both needs to added together, but most derivations I found are not doing that.

Specifically, taking coursers's notation, forward prop:

ft = sigmoid( Wf.(at-1, xt) + bf )
it = sigmoid(Wi.(at-1, xt) + bi )
ctil = Tanh(Wc.(at-1, xt) + bc )

ct = ft * ct-1 + it*ctil
ot = sigmoid(Wo.(at-1, xt) + bo )
at = ot * Tanh(ct)
yhat_t = Softmax(Wy.at + by )

Since at-1 and ot both have Wo in their equations, I would assume following derivation for dWo:

dWo = dat * (1-T(ct)**2 ) * ot* (1-ot )*(at-1,xt)-dot part + dotat-1 * dat-1ot-1 * dot-1Wo

Above derivation contains dot + dot-1. But derivation given by coursera, only contains dot and not dot-1.

And in the derivations of update and forget gates, what follows after plus sign, shouldn't they be t-1 and not t ?

So, I'm assuming LSTM back-prop doesn't involve BPTT , could someone please enlighten me on this ?

I'm specifically looking for the right theoretical derivation of BP in LSTMs, and if they involve BPTT in theory or not.

Any help is very much appreciated.

Thanks !

Coursera LSTM Backprop Derivation

$\endgroup$

1 Answer 1

2
$\begingroup$

In practice you backprop through time. "Backprop through time" is just a fancy way of saying you truncate your input data to a maximum sequence length. Your hidden states backprop to previous timesteps.

For an empirical example, the code below passes an input through a LSTM, backprops through the last hidden state, then inspects the gradient of the input to verify a gradient value is backproped through the input.

import torch
import torch.nn as nn

lstm = nn.LSTM(
    input_size=8,
    hidden_size=16,
    num_layers=3,
    bias=False,
    batch_first=True
)

x = torch.randn(4, 32, 8, requires_grad=True) # (batch_size, sequence_length, input_size)

y, (h, c) = lstm(x)

h.backward(gradient=torch.ones_like(h)) # for example, backprop through hidden state

x.grad.shape
>torch.Size([4, 32, 8])
```
$\endgroup$
1
  • $\begingroup$ Thanks for taking time ! Your explanation on Truncated backprop makes sense.My questions are two fold: 1) If one does grad check with truncated backprop, there is no way grad check is going to pass right ? 2) What is the theoretical derivation of LSTM's backprop , is that similar to RNN's BPTT , or different ? to check if my implementation of LSTM is right, I need to know "2" and then do grad check. I can only move to "1" after I pass grad check. If I can't do grad check, I wouldn't know if my implementation is right. $\endgroup$ Jan 31 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.