TIL that when using the LSTM layer, the states are remembered throughout the same batch. When using stateful LSTM, they can be even remembered outside of the batch.
The first realization gave me a pause though. I often perform NLP tasks on multiple documents cut into chunks (e.g. sentences). Thus, my X matrix (and, consequently, training batches) would often consist of rows that are from the same domain, but are not connected temporally.
My questions are the following:
- If I understand correctly, when LSTM is present, the data are divided into batches first, and the only shuffling that is done between epochs is done on the whole batches? (as opposed to shuffling all rows and then forming batches).
- How big of a problem is this if I have samples from different documents in a batch? It seems that somehow my networks managed to at least mitigate the noise, but perhaps I could get better results were it not for it?
- How to remedy this? I see two potential solutions: either use batch size 1, which will mean no states will be remembered between samples. The second one is to have data arranged in a way that each document is split into n*batch_size chunks. Then I guess every batch would contain only samples. That would mean either truncating sequences, or having some sort of padding-with-samples?
