40
$\begingroup$

How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the number of parameters is useful for training models using LSTMs.

$\endgroup$
27
$\begingroup$

The LSTM has a set of 2 matrices: U and W for each of the (3) gates. The (.) in the diagram indicates multiplication of these matrices with the input $x$ and output $h$.

  • U has dimensions $n \times m$
  • W has dimensions $n \times n$
  • there is a different set of these matrices for each of the three gates(like $U_{forget}$ for the forget gate etc.)
  • there is another set of these matrices for updating the cell state S
  • on top of the mentioned matrices, you need to count the biases (not in the picture)

Hence total # parameters = $4(nm+n^{2} + n)$

LSTM abstract block

$\endgroup$
  • 1
    $\begingroup$ I faced this question myself when taking practical decisions on estimating hardware requirements and project planning for a deep learning project. PS: I didn't answer my own question to just gain reputation points. I want to know if my answer is right from the community. $\endgroup$ – wabbit Mar 9 '16 at 11:17
  • $\begingroup$ You have ignored bias units. See Adam Oudad's answer below. $\endgroup$ – arun Jun 20 '18 at 0:11
  • $\begingroup$ Biases are not there. I have edited the answer. $\endgroup$ – Escachator Oct 20 '18 at 12:09
20
$\begingroup$

Following previous answers, The number of parameters of LSTM, taking input vectors of size $m$ and giving output vectors of size $n$ is:

$$4(nm+n^2)$$

However in case your LSTM includes bias vectors, (this is the default in keras for example), the number becomes:

$$4(nm+n^2 + n)$$

$\endgroup$
  • 1
    $\begingroup$ This is the only complete answer. Every other answer appears content to ignore the case of bias neurons. $\endgroup$ – Sycorax Feb 7 '18 at 14:11
  • $\begingroup$ To give a concrete example, if your input has m=25 dimensions and you use an LSTM layer with n=100 units, then number of params = 4*(100*25 + 100**2 + 100) = 50400. $\endgroup$ – arun Jun 20 '18 at 0:13
  • $\begingroup$ Suppose I am using timestep data, is my understanding below correct? n=100: mean I will have 100 timestep in each sample(example) so I need 100 units. m=25 mean at each timestep, I have 25 features like [weight, height, age ...]. $\endgroup$ – jason zhang Mar 10 at 6:41
  • $\begingroup$ @jasonzhang The number of timesteps is not relevant, because the same LSTM cell will be applied recursively to your input vectors (one vector for each timestep). what arun called "units" is also the size of each output vector, not the number of timesteps. $\endgroup$ – Adam Oudad Mar 11 at 8:24
10
$\begingroup$

According to this:

LSTM cell structure

LSTM cell structure

LSTM equations

LSTM equations

Ingoring non-linearities

Ingoring non-linearities

If the input x_t is of size n×1, and there are d memory cells, then the size of each of W∗ and U∗ is d×n, and d×d resp. The size of W will then be 4d×(n+d). Note that each one of the dd memory cells has its own weights W∗ and U∗, and that the only time memory cell values are shared with other LSTM units is during the product with U∗.

Thanks to Arun Mallya for great presentation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.