Understanding an LSTM network

Question

Hi I was working on a project to generate a hardware implementation of an LSTM network for anomaly detection. I was trying to understand the general structure of an lstm network for forecasting uni-variate time-series. I was working on creating a layer that in Keras would be implemented as

self.model.add(LSTM(units=num_neurons, return_sequences=False, batch_input_shape=(1, b - 1, 1), stateful=True)).

where b is the lookback factor

I was working off the generic image of an lstm cell where it takes in the inputs of x_t, c_t-1, and h_t-1. I think I'm confusing my self with all the different terminology and I was hoping someone could clarify and help me confirm my mental model of how this should all be working.

Starting off the num_neurons in the code, I'm assuming that is the number of lstm cells in layer. I'm assuming the input to each of these cells for the x_t term is the same vector that is a vector of [x, x-1,..,x-b]. I honestly have no clue what c_t-1 and h_t-1 are meant to be. I had two possible ideas. Idea 1 is that each lstm unit in same layer feeds into the next one. Idea 2 is that each lstm in same layer are not connected at all rather the layer is iterated through several times, like each cells output feeds back into itself several times. In either case, I don't know what the base case of h and c would be.

I found this one youtube video by StatQuest which honestly makes perfect sense, https://www.youtube.com/watch?v=YCzL96nL7j0. The lstm network described is a series of connected LSTM cells that each take in one value each from the vector [x_t, x_t-1, ... x_t-b] and this makes complete sense to me. But then I don't understand how in keras you can have a different number of neurons and a different input vector size.

I drew up this image of the model that I think is how an lstm works. Could someone confirm this is on the right track or if its just completely wrong?

Answer 1

I don't know what the base case of h and c would be.

The initial values for the cell state and previous output are generally 0.

If you are confused with what $c$ and $h$ actually are, $h_{t-1}$ represents the previous neuron's output, which is sort of like its short term memory. and $c_{t-1}$ represents the previous neuron's cell state, which is sort of like its long term memory, hence: long short term memory.

I don't understand how in keras you can have a different number of neurons and a different input vector size.

I think you are confusing time steps with inputs. Keep in mind values like $x_t$ represent entire vectors that are passed into each neuron at that time step. This means that you can pass an input vector with size 26 into 16 LSTM neurons.

PS: It is weird to refer to the "input vector" as a vector of all the input vectors at each time step, which is probably where your confusion stemmed from. The input vector is a vector of each input at a certain time step.

As for the diagram...

It is almost correct. When you drew the lines connecting from the input to each neuron, I hope you understand that those lines represent entire vectors of inputs being passed to each neuron. Also, you may have fallen for the misconception that at each time step, each LSTM neuron updates independently. As I have seen you have bordered each LSTM neuron with a box. This is wrong. Just like how an entire input vector gets passed into each LSTM neuron, each of the LSTM neuron's previous outputs get passed into each LSTM neuron. This means that the previous output in the top LSTM neuron can affect the bottom LSTM neuron's output in the next time step.