Understanding LSTM Keras implementation

Question

So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework.

Let's say, I add a layer

model.add(LSTM(5, ...))

does this mean, I have

• five different LSTM units, that are independent, or
• one LSTM unit, that is unfolded into five time steps?

I assume that the second answer is right, because else a lot of models I saw would not make much sense.

When I have as first hidden layer a LSTM layer with size 5, does this mean my input should be a time series with 5 time steps?

Please correct me if I'm wrong. I am confused because I thought one LSTM would use its own state in the next iteration.

What is the correct meaning of LSTM in Keras?

Answer 1

I am afraid both of your description is incorrect but the first one is the closest.

Each cell/timestep will actually produce a hidden state representation. However, this is surpressed by default on keras. You can re. As LSTM without Bidirectional wrapper produce a one-way sweep through your sequence, your output from LSTM(5) is the hidden state of the final LSTM cell(cell output not cell state).

Now to enable outputting the sequence of hidden output similar to what you might have learned about LSTM, simply supply return_sequence=True.

To compare output shape:

Without return_sequence=True, Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,hidden_dim]

With return_sequence=True, Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,sequence_length,hidden_dim]

What you are referring as "folding" of LSTM is actually iautomatically based on sequence length of previous input.

In case of any doubt on my explanation, please read the material I give you below. You can verify my description by calling model.summary().

So basically what led to your confusion is due to keras LSTM default behaviour of only outputting the final cell output.

LSTM link