0
$\begingroup$

The following is mentioned in the official keras RNN documentation (https://www.tensorflow.org/guide/keras/rnn). By "Normally", I assume they mean when stateful=False, which is also the default setting for this parameter.

Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assumed to be independent of the past). The layer will only maintain a state while processing a given sample.

Question

The meaning of the above text is not clear to me, because the bold parts seem to provide conflicting information. By sample, do they mean a single sequence (i.e. a frame of (timesteps x features) elements to which a label has been given), or an entire batch of sequences? Also, which of the following is true:

(inter-batch reset): The state will reset only after each batch, because the text clearly states "is reset every time it sees a new batch". This would mean that state is maintained within each batch. Thus, if batch_size = 32, the hidden state produced by the 1st sequence should be initial state for 2nd sequence, and so forth until the hidden state produced by the 31th sequence is initial state for 32th. Then, the state is reset for the next batch.

(intra-batch reset): The state will reset after each sequence, because the text mentions that "The layer will only maintain a state while processing a given sample". Thus, all sequences are independent with each other, even if they happen to be in the same batch. I suspect that this is the right answer, but then why didn't they simply write:

Normally, the internal state of a RNN layer is reset every time it sees a new sequence ...

instead of

Normally, the internal state of a RNN layer is reset every time it sees a new batch ...

$\endgroup$

2 Answers 2

1
+50
$\begingroup$

First, let's clarify some things:

  • Each sequence in a batch is totally independent of the rest of the sequences from the same batch. This behavior is fixed and cannot be changed.
  • The initial state is actually a batch of initial states, where each initial state in the batch applies to the associated sequence in the input batch.
  • You normally want to supply the initial state to the RNN under three different circumstances:
    • When you want to initialize it to zeros.
    • When a different part of the model computes it. This happens, for instance, in encoder-decoder architectures without attention, where the initial state of the decoder is the final state of the encoder.
    • When you want to implement truncated back-propagation through time. This means that your data presents dependencies to past values that are farther away than the training batch maximum sequence length, but you don't want to back-propagate that much. In such cases, you mark your RNN as "stateful" and then it uses the state of the last batch as the initial state of the subsequent batch. For this to actually be useful, the batches have to be prepared in a way that the ending of the sequences in the last batch matches with the beginning of the sequences in the subsequent batch. This answer explains more details about the stateful flag in LSTMs.

Now, the answer:

  • "sample" here means sequence. All samples in a batch are always independent of one another.
  • by default, the initial state is reset at every batch; this means that each sequence in the batch is computed independently from the sequence in the same position of the previous batch. You can change the behavior of the RNN to make it use as initial state the last state of the previous batch, meaning that the initial state for each sequence of the new batch is the last state of the sequence at the same position within the previous batch.
  • If, as you suggested, they would have written "Normally, the internal state of a RNN layer is reset every time it sees a new sequence ..." (instead of saying "batch"), it would be very confusing, because a single batch contains multiple sequences, and you just can't keep the state across sequences in the same batch so using "normally" would be incorrect there.
$\endgroup$
2
  • $\begingroup$ Thank you. Specifically, this part "The initial state is actually a batch of initial states." is the part I was missing. Is this always true (e.g. with CPU mode)? Also, you write: "you just can't keep the state across sequences in the same batch". Can you provide explanation for this e.g. is it a design limitation of keras? Does it have to do with the fact that batch computation is executed in parallel with GPUs (so that all sequences must be executed simultaneously), or am I way off? Finally, please provide any sources or references that you recommend for further details. $\endgroup$
    – Enk9456
    Apr 23, 2022 at 16:32
  • 1
    $\begingroup$ 1. About, "The initial state is actually a batch of initial states.": yes, it is always true. 2. About "you just can't keep the state across sequences in the same batch": this is by design. If you want to do use the final state of a sequence in a batch as the initial state of the next sequence in the same batch, you can simply concatenate all the sequences in the batch. $\endgroup$
    – noe
    Apr 23, 2022 at 16:36
2
$\begingroup$

Internal state of RNN is reset every time it sees a new batch. The layer will only maintain the state while processing the samples in a batch. If you think logically if a model resets its internal state everytime it sees a new sample it would not be able to learn properly and will not give good results

If you want to maintain state for all the batch please use

stateful=True in the contructor

this has been demonstrated with example in the link below: References :Cross Batch Statefullness

$\endgroup$
1
  • $\begingroup$ What is the meaning of "Every sample seen by the layer is assumed to be independent of the past". Furthermore, if one batch has more than samples (e.g. 32), wouldn't that mean that state resets take place inside the batch, in order for the samples to be independent? $\endgroup$
    – Enk9456
    Apr 18, 2022 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.