Why are the hidden states of an RNN initialised every epoch instead of every batch?

Question

Why are the hidden states of RNNs/LSTMs/GRUs generally re-initialised only once an epoch has finished, not once a batch has finished?

Answer 1

Hidden states of an RNN are initialized every epoch ONLY if you are using truncated back-propagation through time (TBPTT). If you are not using TBPTT, then you would reset the RNN hidden states at every batch.

In the TBPTT training approach, the gradient is propagated through the hidden states of the LSTM across the time dimension in the batch and then, in the next batch, the last hidden states are used as input states for the LSTM. This allows the LSTM to use longer context at training time while constraining the number of steps back for the gradient computation.

As you are reusing hidden states from one batch to the following, you don't want to reset them every batch.

Usually, deep learning frameworks provide some sort of flag to have "stateful" hidden states, enabling this way TBPTT if the data preparation logic supports it (you need to ensure that successive batches actually contain "successive data" for TBPTT to make sense).

I know of two scenarios where using TBPTT is common:

• Language modeling (LM).
• Time series modeling.

The training set is a list of sequences, potentially coming from a few documents (LM) or complete time series. During data preparation, the batches are created so that each sequence in a batch is the continuation of the sequence at the same position in the previous batch. This allows having document-level/long time series context when computing predictions.

In these cases, your data is longer than the sequence length dimension in the batch. This may be due to constraints in the available GPU memory (therefore limiting the maximum batch size) or by design due to any other reasons.

Note: I reused parts of another one of my answers that explained what the flag stateful = True was about in Keras' LSTMs.