0
$\begingroup$

When using tf.keras.layers.SimpleRNN,does this SimpleRNN have a hidden state, or does it just use the output value as the hidden state. That is, does it follow the formulas $h_t = \tanh(w_h\cdot h_{t-1} + w_x\cdot x_{t-1}+b_h)$, $y_t = w_o\cdot h_t + b_o$? Or is $h_t=y_t$?

If the input is of length 5, and I make keras.layers.SimpleRNN(1), why are there only 6 parameters according to the summary. There should be 5 for $w_x$, 1 for $b_h$, 1 for $w_h$, and possibly 2 for output, for a total to 7 or 9.

$\endgroup$

2 Answers 2

0
$\begingroup$

From my experience and from this article it appears that SimpleRNN simply uses the hidden state as output. It does not compute any separate output. This is not stated explicitly in the official documentation.

Using the code below you can see it for yourself that the state and output are always the same.

inputs = np.random.random([2, 3, 1]).astype(np.float32)
simple_rnn = tf.keras.layers.SimpleRNN(4, return_state=True)

output, state = simple_rnn(inputs)

print(output)
print("====")
print(state)
$\endgroup$
0
$\begingroup$

Note: In Keras, every SimpleRNN has only three different weight matrices, and these weights are shared between all input cells; In other words, for all five cells in your network: \begin{align} h_t = tanh( w_{h} h_{t-1} + w_{x} x_{t} + b_h)\ ; t= 1..5 \end{align} For a deeper understanding of recurrent networks in Keras, you may want to read this eloquent article: Mohit Mayank - A practical guide to RNN and LSTM in Keras.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.