The following is mentioned in the official keras RNN documentation (https://www.tensorflow.org/guide/keras/rnn). By "Normally", I assume they mean when stateful=False, which is also the default setting for this parameter.
Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assumed to be independent of the past). The layer will only maintain a state while processing a given sample.
The meaning of the above text is not clear to me, because the bold parts seem to provide conflicting information. By sample, do they mean a single sequence (i.e. a frame of (timesteps x features) elements to which a label has been given), or an entire batch of sequences? Also, which of the following is true:
(inter-batch reset): The state will reset only after each batch, because the text clearly states "is reset every time it sees a new batch". This would mean that state is maintained within each batch. Thus, if batch_size = 32, the hidden state produced by the 1st sequence should be initial state for 2nd sequence, and so forth until the hidden state produced by the 31th sequence is initial state for 32th. Then, the state is reset for the next batch.
(intra-batch reset): The state will reset after each sequence, because the text mentions that "The layer will only maintain a state while processing a given sample". Thus, all sequences are independent with each other, even if they happen to be in the same batch. I suspect that this is the right answer, but then why didn't they simply write:
Normally, the internal state of a RNN layer is reset every time it sees a new sequence ...
Normally, the internal state of a RNN layer is reset every time it sees a new batch ...