Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps

Question

I am building an LSTM to attempt to learn the historic trend of some time-series data set (e.g. the daily share price of a company). When training my network, I am taking batches of size 1, each consisting of 25 sequential daily closing prices, where it then makes a prediction for the price on the 26th day.

Say the first sequence in an epoch used to train begins at $t=0$ and ends at $t=24$. I am using a stateful LSTM and hence taking the output state of one batch and inputting it to the state of the next batch, does this mean that my 2nd batch must be $[t=25, t=49]$?

Can I instead "slide" each batch by 1 time step so that the 2nd batch is $[t=1, t=25]$, or does this defeat the purpose of passing the state between batches?

Answer 1

The second batch should be [t=25,t=49] because a Stateful LSTM must be sequential, because as you rightly say, the state of the network is at the state of t=24, if you then feed t=2 (or anything other than t=25) then the sequencing is messed up.

With stateless LSTM you can create sequences from any starting point, but with stateless LSTM the sequence must be observed.

In subsequent epochs you could change your start index so that different sequences are inputted. Nothing wrong with that. I personally generate a random number between 0 and the sequence length.