TIL that when using the LSTM layer, the states are remembered throughout the same batch. When using stateful LSTM, they can be even remembered outside of the batch.

The first realization gave me a pause though. I often perform NLP tasks on multiple documents cut into chunks (e.g. sentences). Thus, my X matrix (and, consequently, training batches) would often consist of rows that are from the same domain, but are not connected temporally.

My questions are the following:

  1. If I understand correctly, when LSTM is present, the data are divided into batches first, and the only shuffling that is done between epochs is done on the whole batches? (as opposed to shuffling all rows and then forming batches).
  2. How big of a problem is this if I have samples from different documents in a batch? It seems that somehow my networks managed to at least mitigate the noise, but perhaps I could get better results were it not for it?
  3. How to remedy this? I see two potential solutions: either use batch size 1, which will mean no states will be remembered between samples. The second one is to have data arranged in a way that each document is split into n*batch_size chunks. Then I guess every batch would contain only samples. That would mean either truncating sequences, or having some sort of padding-with-samples?


  • $\begingroup$ I dont see the point in this question. Obviously if you use sequences from different documents to train the same LSTM you obviously assume that these are related. Else use different LSTM instances for each document. $\endgroup$
    – Nikos M.
    Sep 21, 2021 at 11:13
  • $\begingroup$ Documents are related but not in a sequential way. What I would envision is that the state is saved for the instances from the same document (group of samples), and reset once a new document is reached. $\endgroup$
    – JohnnyQ
    Sep 21, 2021 at 11:35
  • $\begingroup$ Imagine having an LSTM layer in a text generator. In training data you will have many different documents that are related linguistically, but not temporally. Sentences from each document are related temporally though. You wouldn't have a different model for each document here. $\endgroup$
    – JohnnyQ
    Sep 21, 2021 at 11:37
  • 1
    $\begingroup$ I understand, this might be an issue, in any case I think there is an option to manually reset LSTM states, look it up $\endgroup$
    – Nikos M.
    Sep 21, 2021 at 11:40
  • 1
    $\begingroup$ this might help $\endgroup$
    – Nikos M.
    Sep 21, 2021 at 11:42


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.