Why not Back propagate through time in LSTM , similar to RNN

Question

I'm trying to implement RNN and LSTM , many-to-many architecture. I reasoned myself why BPTT is necessary in RNNs and it makes sense.

But what doesn't make sense to me is, most of resources I went through online for LSTM back prop (like attachment ) and this one : https://www.youtube.com/watch?v=8rQPJnyGLlY

Seems to be doing more of back propagation w.r.t current time stamp itself, but not through time , keeping variable convention aside, similar to RNN , I would assume that when calculating DL w.r.t Wf, since both current time step and previous hidden states are made up of Wf, both needs to added together, but most derivations I found are not doing that.

Specifically, taking coursers's notation, forward prop:

ft = sigmoid( Wf.(at-1, xt) + bf )
it = sigmoid(Wi.(at-1, xt) + bi )
ctil = Tanh(Wc.(at-1, xt) + bc )

ct = ft * ct-1 + it*ctil
ot = sigmoid(Wo.(at-1, xt) + bo )
at = ot * Tanh(ct)
yhat_t = Softmax(Wy.at + by )

Since at-1 and ot both have Wo in their equations, I would assume following derivation for dWo:

dWo = dat * (1-T(ct)**2 ) * ot* (1-ot )*(at-1,xt)-dot part + dotat-1 * dat-1ot-1 * dot-1Wo

Above derivation contains dot + dot-1. But derivation given by coursera, only contains dot and not dot-1.

And in the derivations of update and forget gates, what follows after plus sign, shouldn't they be t-1 and not t ?

So, I'm assuming LSTM back-prop doesn't involve BPTT , could someone please enlighten me on this ?

I'm specifically looking for the right theoretical derivation of BP in LSTMs, and if they involve BPTT in theory or not.

Any help is very much appreciated.

Thanks !

Answer 1

In practice you backprop through time. "Backprop through time" is just a fancy way of saying you truncate your input data to a maximum sequence length. Your hidden states backprop to previous timesteps.

For an empirical example, the code below passes an input through a LSTM, backprops through the last hidden state, then inspects the gradient of the input to verify a gradient value is backproped through the input.

import torch
import torch.nn as nn

lstm = nn.LSTM(
    input_size=8,
    hidden_size=16,
    num_layers=3,
    bias=False,
    batch_first=True
)

x = torch.randn(4, 32, 8, requires_grad=True) # (batch_size, sequence_length, input_size)

y, (h, c) = lstm(x)

h.backward(gradient=torch.ones_like(h)) # for example, backprop through hidden state

x.grad.shape
>torch.Size([4, 32, 8])
```