6
$\begingroup$

I paste below the Keras documentation on Recurrent layer

model= Sequential()
model.add(LSTM(32, input_shape=(10, 64)))

# now model.output_shape == (None, 32)
# note: `None` is the batch dimension.

What does the first argument of LSTM 32 mean here? Is it creating 32 LSTM blocks (By block I mean consisting of input, forget and output gate)? Could you please explain the meaning of the first argument and how that contributes to the output dimension of the LSTM both when return sequences is True and when return sequences is False

$\endgroup$
6
$\begingroup$

The notation used for LSTM is quite confusing, and this took me some time to get my head around this as well. When you see this graphic (often used to explain RNNs):

enter image description here

You need to consider that X is a sequence of data the timestep t, it's not just a single scalar input value (as we're used to with feed-forward networks); it's an array / tensor of data. The diagram shows how there is an output at time-step t, but that it then also feeds into the next time-step, t+1, when the next array/tensor is then fed in.

A better/clearer way (in my opinion) is to look at it is like this:

enter image description here

So a LSTM 'cell' is actually what you might consider a layer. And a unit (one of the circles) is one of these:

enter image description here

which you can consider a neuron in a hidden layer.

Initially, I thought this was a cell, but it isn't, it's a single unit. So when you specify 32 units, for example, you're actually saying how many of these units (neurons) you want in the cell (layer).

This is what gives the model its capacity to learn the data that's presented to it. And just like hidden layers/neurons in a feed-forward, it's a hyperparameter that you'll need to experiment with; too low and the model will under-fit, too high and it will over-fit.

|improve this answer|||||
$\endgroup$
1
$\begingroup$

The value '32' in this case is the size of the cell state and the size of the hidden state being sent forward in the network. Please see my answer here for more information.

|improve this answer|||||
$\endgroup$
0
$\begingroup$

The question may be too old but I think the BigBadMe answer is not true. As the keras docs said:

units: Positive integer, dimensionality of the output space.

The number of units actually is the dimension of the hidden state (or the output).

For example, in the image below, the hidden state (the red circles) has length 2. The number of units is the number of neurons connected to the layer holding the concatenated vector of hidden state and input (the layer holding both red and green circles below). In this example, there are 2 neurons connected to that layer.

The number of units defines the dimension of hidden states (or outputs) and the number of params in the LSTM layer.

enter image description here

The image above is taken from: https://towardsdatascience.com/counting-no-of-parameters-in-deep-learning-models-by-hand-8f1716241889, which is a helpful article on LSTM

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.