Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT

Question

What I'm trying to do

What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory the series is infinity. If a new value $x$ is coming, i feed it into the network and predict the next one. So seq-to-seq model with the return_sequenze parameter set to $True$ does not work, because in practical application, at time $t$ we do not have the data-points $x_i$ for $i>t$. For a multistep-prediction I use a manual loop back, where I feed the predicted value $\hat{x}$ as the new input.(Knowing what comes next - compensation if nothing comes)

My understanding of LSTM

Maybe there is something basic that I have misunderstood :

Timelags

If we take a stepsize of 3 (in a one-to-many network), we are ending up with input/output samples $([x_0,x_1,x_2], x_3),([x_1,x_2,x_3], x_4), ..., ([x_{T-3},x_{T-2},x_{T-1}], x_T)$ (Using Rolling-windows), or $([x_0,x_1,x_2], x_3),([x_3,x_4,x_5], x_6), ..., ([x_{T-3},x_{T-2},x_{T-1}], x_T)$ (Not using Rolling-windows). Using a batchsize of 1, the networkinput is $(1,3,n\_features)$, where $n\_features$ is the dimension of $X_i$ ($1$ for an univariat timeseries, $>1$ for multivariate timeseries).

Batches and timelags

So far so good. But, considering the batchsize and that within the batch the states of the Cells are not reset, the initial state of the second sample is the state at the end of the first sample. So with a batchsize of $2$ the network gets the input as shown in the picture below (Inputshape would be $(2, 3, n\_features)$).The number in the superscript indicates the sample. Doing so, we are messing up the timeseries, because, as you can see, the series now is $[x_0, x_1, x_2, x_1, x_2, x_3]$. (The same problem occurs between to batches, if the network is set to be stateful). If the samples are not generated by rolling-windows, its obvious that there are missing values in the output sequence ($y_4$ and $y_6$ would be the outputs). And since there is no useful reason to do a seq-to-seq prediction for my problem (as stated previous), I'll ignore that case for now.

As there is a good reason to use batches to train RNNs (there is an article at machinelearningmastery) and not to update the weights after every sample, I am using a single timestep for each sample. The inputshape is now $(batch\_size, 1, n\_features)$. For example using a batchsize of 6 the networks calculation should look like this. (For stateful networks the next observation would be $x_6, ...$ together with the state at $x_5$, the batchsize determines when to update the weights during training.)

Truncated Backpropagation Through Time

As stated in this stackoverflow question the BPTT for keras RNNs is limited to the timelags of the input. Lets have a look at our input shape considering the BPTT. For a higher order of lags, the error is back-propagated through time.

For the approach, where the input is $(batch\_size, 1, n\_features)$ there is basically no BPTT, because it stops at one. And I think, that the weights of the recurrent connection are not updated at all, are they?

Conclusion and Question

Learning with a timelag of one might not be a good idea (mainly because of the insufficient BPTT, as the states can be preserved). Working with a higher order of timelags leads into a) holes in the output data using no rolling-windows while generating the samples or b) the time sequence is messed up.

What is a good approach to archive BPTT with Keras LSTMs, predicting future values (one-step and recursive multistep prediction as will) and still keep the input data stream-like (one observation after another)? Does some one have experienced similar problems and if so, how did you solve it?

I'm thankful for every input and maybe I also could help someone who is working on similar stuff ;)

My theoretical solution

While writing this question, I had this idea:

As far as i understand the Keras Layers, I can use TimeDistributed Outputlayer to have the Outputlayer applied to every given lag and the networks returns the sequence. (seq-to-seq network)

input = Input(batch_shape=(batch_size, timelags, features))
lstm = LSTM(units=10, return_sequences=True)(input)
out = TimeDistributed(Dense(units=features))(lstm)

To avoid messing up the sequence, I do not use a rolling-window. The Input/Outputs are (for lags=3) $([x_0,x_1,x_2], [x_1, x_2, x_3]),([x_3, x_4, x_4],[x_4, x_5, x_6])...)$ After I have trained the model I build a new one, which has the same architecture, but the inputshpape $(1, 1, features)$ and copy the trained parameters. This works, because the each LSTM- and outputpart as shown in the image are the actually the same (it shows the unrolled version of the lstm) and there is no need for BPTT while predicting, so no need to keep timelags > one and as the network is stateful the hiddenstates are preserved and the output should be the same as if I use a higher order of timelags with a sequence as output.

prediction_model.set_weights(trained_model.get_weights())

This way the BPTT is used and the input can be fed as stream of single points in time while predicting(I'm testing it now).

Answer 1

There are several approaches that you can use to achieve the type of prediction you are describing with a Keras LSTM model. One approach is to use a sliding window to generate your training and prediction sequences, as you mentioned in your question. This involves splitting the input sequence into overlapping segments of a fixed length (e.g. three timesteps), and using each segment as an input to the model to predict the next timestep. This allows you to train the model on multiple timesteps at once and still make one-step predictions in real-time.

Another approach is to use a stateful LSTM model, which allows the state of the network to be preserved between batches. This allows you to train the model on a sequence of input data, and then use the final state of the network to make a one-step prediction. You can then feed the predicted value back into the network as the next input, and repeat the process to make multiple predictions in sequence.

In both cases, it is important to use a sufficiently large batch size when training the model to ensure that the state of the network is properly updated at each timestep. You can also experiment with different timestep lengths and overlapping window sizes to find the best configuration for your specific dataset and application.