Stateful LSTM : Using different training window

Question

Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all of the data for that epoch I then call model.reset_states()

After that epoch, could I then move the window forward by some arbitrary amount e.g. 2, so I then feed in [2-11],[12-21],[22-31],[32-41]...[982-991]

I would do that for 99 batches (the final sequence in the batch is now length 8 so I can't make a complete sequence).

Would doing that make sense? That way the network learns sequences from different starting/end points, and has differing output/Y values to train against.

Answer 1

I don’t see anything wrong with that. In fact it sounds like a good form of data augmentation.

It does sound like you are training with a batch size of 1, which may be slow. You could think about creating batches in a similar way, where the first elements of the batches are the sequences 1-10, 11-20, ..., the second elements of the batches are 2-12, 13-23, ..., all the way to 9-19, 10-29, ...