Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all of the data for that epoch I then call model.reset_states()

After that epoch, could I then move the window forward by some arbitrary amount e.g. 2, so I then feed in [2-11],[12-21],[22-31],[32-41]...[982-991]

I would do that for 99 batches (the final sequence in the batch is now length 8 so I can't make a complete sequence).

Would doing that make sense? That way the network learns sequences from different starting/end points, and has differing output/Y values to train against.


I don’t see anything wrong with that. In fact it sounds like a good form of data augmentation.

It does sound like you are training with a batch size of 1, which may be slow. You could think about creating batches in a similar way, where the first elements of the batches are the sequences 1-10, 11-20, ..., the second elements of the batches are 2-12, 13-23, ..., all the way to 9-19, 10-29, ...

| improve this answer | |
  • $\begingroup$ Thanks kbrose. I've implemented this method and seems to work well. Yes at the time of writing I was using batch size of 1 because I have multi-series data, which was of varying lengths, so I was using bs=1 to enable all data to be used. But now I just discard a series if I can't make up a bigger batch size. I figure if the series was so small then a stateful LSTM wouldn't be able to build much state anyway for it to be useful. It may even be detrimental to the learning. $\endgroup$ – BigBadMe Aug 3 '18 at 19:13
  • $\begingroup$ Could I ask a favour; I'm new to LSTM and have a few unanswered questions on the topic. Would you mind looking to see if there's any other questions you might know the answer to: datascience.stackexchange.com/users/51125/… $\endgroup$ – BigBadMe Aug 3 '18 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.