Keras: ambiguity regarding state maintenance in RNNs

Question

The following is mentioned in the official keras RNN documentation (https://www.tensorflow.org/guide/keras/rnn). By "Normally", I assume they mean when stateful=False, which is also the default setting for this parameter.

Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assumed to be independent of the past). The layer will only maintain a state while processing a given sample.

Question

The meaning of the above text is not clear to me, because the bold parts seem to provide conflicting information. By sample, do they mean a single sequence (i.e. a frame of (timesteps x features) elements to which a label has been given), or an entire batch of sequences? Also, which of the following is true:

(inter-batch reset): The state will reset only after each batch, because the text clearly states "is reset every time it sees a new batch". This would mean that state is maintained within each batch. Thus, if batch_size = 32, the hidden state produced by the 1st sequence should be initial state for 2nd sequence, and so forth until the hidden state produced by the 31th sequence is initial state for 32th. Then, the state is reset for the next batch.

(intra-batch reset): The state will reset after each sequence, because the text mentions that "The layer will only maintain a state while processing a given sample". Thus, all sequences are independent with each other, even if they happen to be in the same batch. I suspect that this is the right answer, but then why didn't they simply write:

Normally, the internal state of a RNN layer is reset every time it sees a new sequence ...

instead of

Normally, the internal state of a RNN layer is reset every time it sees a new batch ...

Answer 1

First, let's clarify some things:

• Each sequence in a batch is totally independent of the rest of the sequences from the same batch. This behavior is fixed and cannot be changed.
• The initial state is actually a batch of initial states, where each initial state in the batch applies to the associated sequence in the input batch.
• You normally want to supply the initial state to the RNN under three different circumstances:
  • When you want to initialize it to zeros.
  • When a different part of the model computes it. This happens, for instance, in encoder-decoder architectures without attention, where the initial state of the decoder is the final state of the encoder.
  • When you want to implement truncated back-propagation through time. This means that your data presents dependencies to past values that are farther away than the training batch maximum sequence length, but you don't want to back-propagate that much. In such cases, you mark your RNN as "stateful" and then it uses the state of the last batch as the initial state of the subsequent batch. For this to actually be useful, the batches have to be prepared in a way that the ending of the sequences in the last batch matches with the beginning of the sequences in the subsequent batch. This answer explains more details about the stateful flag in LSTMs.

Now, the answer:

• "sample" here means sequence. All samples in a batch are always independent of one another.
• by default, the initial state is reset at every batch; this means that each sequence in the batch is computed independently from the sequence in the same position of the previous batch. You can change the behavior of the RNN to make it use as initial state the last state of the previous batch, meaning that the initial state for each sequence of the new batch is the last state of the sequence at the same position within the previous batch.
• If, as you suggested, they would have written "Normally, the internal state of a RNN layer is reset every time it sees a new sequence ..." (instead of saying "batch"), it would be very confusing, because a single batch contains multiple sequences, and you just can't keep the state across sequences in the same batch so using "normally" would be incorrect there.

Answer 2

Internal state of RNN is reset every time it sees a new batch. The layer will only maintain the state while processing the samples in a batch. If you think logically if a model resets its internal state everytime it sees a new sample it would not be able to learn properly and will not give good results

If you want to maintain state for all the batch please use

stateful=True in the contructor

this has been demonstrated with example in the link below: References :Cross Batch Statefullness