0
$\begingroup$

Why does this code...

model = tf.keras.Sequential()   
model.add(tf.keras.layers.LSTM(9,input_shape=(6,1),activation='relu'))
print(model.summary())

... print 396 parameteres for the neural network?

I mean, which calculation gave this number, and not more or not less? I would like to understand a bit of the logic behind what I'm doing.

$\endgroup$

1 Answer 1

2
$\begingroup$

You can look at the individual weights via:

[i.shape for i in model.get_weights()]
> [(1, 36), (9, 36), (36,)]

Your LSTM has an input size of 1 and a hidden size of 9. LSTM implementations pack the input, forget gate, cell gate, and output gate into a single matrix, so the 36 comes from 4*9 weights packed together.

  1. The first weight, of shape (1, 36), is the "input to hidden" weight matrix.
  2. The second weight, of shape (9, 36), is the "hidden to hidden" weight matrix.
  3. The final weight, of shape (36,), is the bias vector.

1*36 + 9*36 + 36 = 396

You can read more about the underlying equations of the LSTM module here.

$\endgroup$
4
  • $\begingroup$ How does 36 becomes 396? $\endgroup$ Apr 15 at 19:17
  • 1
    $\begingroup$ 396 is the total parameter count. 1*36 + 9*36 + 36 = 396 $\endgroup$
    – Karl
    Apr 15 at 19:23
  • $\begingroup$ About "4*9", the text you linked really mentions that "Instead of having a single neural network layer, there are four, interacting in a very special way." But, which are these four layers? Do they have names or descriptions? The text don't say anything about them. $\endgroup$ Apr 16 at 14:06
  • 2
    $\begingroup$ There's the input gate, forget gate, cell gate and output gate. They aren't four layers, but four weight matrices. For efficiency they are packed into a single tensor $\endgroup$
    – Karl
    Apr 16 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.