38
$\begingroup$

From Tensorflow code: Tensorflow. RnnCell.

num_units: int, The number of units in the LSTM cell.

I can't understand what this means. What are the units of LSTM cell? Input, Output and Forget gates? Does this mean "the number of units in the recurrent projection layer for Deep LSTM"? Then why is it called "number of units in the LSTM cell"? What is an LSTM cell and how is it different from an LSTM block, what is the minimal LSTM unit if not a cell?

Improve this question
$\endgroup$
2
  • $\begingroup$ FYI Understanding LSTM units vs. cells $\endgroup$
    – Franck Dernoncourt
    Mar 7, 2017 at 21:44
  • $\begingroup$ If the layer contains multiple parallel LSTM units, how it handles the input x? case 1: h(1) = f(x) h(t) = f(h(t-1),x) h(t+1) = f(h(t), x) case 2: h(1) = f(x) h(t) = h(t-1) Maybe there is other case like ResNet. $\endgroup$
    – user1908842
    Apr 22, 2017 at 12:54

4 Answers 4

Reset to default
16
$\begingroup$

As the helpful comments in that function say,

The definition of cell in this package differs from the definition used in the literature. In the literature, cell refers to an object with a single scalar output. The definition in this package refers to a horizontal array of such units.

In essence, the layer will contain multiple parallel LSTM units, structurally identical but each eventually "learning to remember" some different thing.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Thanks :) That comment was added 7 days ago, after this question. After some digging I have asked Tensorflow team in Google group why they definition of LSTM cell is differs from literature LSTM cell.. and they have added that comment :) $\endgroup$
    – Brans Ds
    Aug 3, 2016 at 17:59
22
$\begingroup$

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion. Each hidden layer has hidden cells, as many as the number of time steps. And further, each hidden cell is made up of multiple hidden units, like in the diagram below. Therefore, the dimensionality of a hidden layer matrix in RNN is (number of time steps, number of hidden units).

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is the clearest answer that I have seen so far! $\endgroup$
    – Jinhua Wang
    Mar 5, 2020 at 20:40
  • $\begingroup$ Ok, but what about features? Isn't it (number of time steps, number of features, number of hidden units)? $\endgroup$
    – Maximus
    May 23, 2020 at 23:03
3
$\begingroup$

In Keras, which sits on top of either TensorFlow or Theano, when you call model.add(LSTM(num_units)), num_units is the dimensionality of the output space (from here, line 863). To me, that means num_units is the number of hidden units whose activations get sent forward to the next time step.

Improve this answer
$\endgroup$
1
$\begingroup$

The number of units in a RNN is the number of RNN memory units to each input of the sequence in vertical manner attached to each other, and each one is passing the filtered information to next memory units.

Remember this concept was inspired from the computer science concept of the flow of memory allocation to different units the bit size.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.