Clarification on the Keras Recurrent Unit Cell

Question

I paste below the Keras documentation on Recurrent layer

model= Sequential()
model.add(LSTM(32, input_shape=(10, 64)))

# now model.output_shape == (None, 32)
# note: `None` is the batch dimension.

What does the first argument of LSTM 32 mean here? Is it creating 32 LSTM blocks (By block I mean consisting of input, forget and output gate)? Could you please explain the meaning of the first argument and how that contributes to the output dimension of the LSTM both when return sequences is True and when return sequences is False

Answer 1

The notation used for LSTM is quite confusing, and this took me some time to get my head around this as well. When you see this graphic (often used to explain RNNs):

You need to consider that X is a sequence of data the timestep t, it's not just a single scalar input value (as we're used to with feed-forward networks); it's an array / tensor of data. The diagram shows how there is an output at time-step t, but that it then also feeds into the next time-step, t+1, when the next array/tensor is then fed in.

A better/clearer way (in my opinion) is to look at it is like this:

So a LSTM 'cell' is actually what you might consider a layer. And a unit (one of the circles) is one of these:

which you can consider a neuron in a hidden layer.

Initially, I thought this was a cell, but it isn't, it's a single unit. So when you specify 32 units, for example, you're actually saying how many of these units (neurons) you want in the cell (layer).

This is what gives the model its capacity to learn the data that's presented to it. And just like hidden layers/neurons in a feed-forward, it's a hyperparameter that you'll need to experiment with; too low and the model will under-fit, too high and it will over-fit.

Answer 2

The value '32' in this case is the size of the cell state and the size of the hidden state being sent forward in the network. Please see my answer here for more information.

Answer 3

The question may be too old but I think the BigBadMe answer is not true. As the keras docs said:

units: Positive integer, dimensionality of the output space.

The number of units actually is the dimension of the hidden state (or the output).

For example, in the image below, the hidden state (the red circles) has length 2. The number of units is the number of neurons connected to the layer holding the concatenated vector of hidden state and input (the layer holding both red and green circles below). In this example, there are 2 neurons connected to that layer.

The number of units defines the dimension of hidden states (or outputs) and the number of params in the LSTM layer.

The image above is taken from: https://towardsdatascience.com/counting-no-of-parameters-in-deep-learning-models-by-hand-8f1716241889, which is a helpful article on LSTM