What I'm trying to do
What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory the series is infinity. If a new value $x$ is coming, i feed it into the network and predict the next one. So seq-to-seq model with the return_sequenze parameter set to $True$ does not work, because in practical application, at time $t$ we do not have the data-points $x_i$ for $i>t$. For a multistep-prediction I use a manual loop back, where I feed the predicted value $\hat{x}$ as the new input.(Knowing what comes next - compensation if nothing comes)
My understanding of LSTM
Maybe there is something basic that I have misunderstood :
Timelags
If we take a stepsize of 3 (in a one-to-many network), we are ending up with input/output samples $([x_0,x_1,x_2], x_3),([x_1,x_2,x_3], x_4), ..., ([x_{T-3},x_{T-2},x_{T-1}], x_T)$ (Using Rolling-windows), or $([x_0,x_1,x_2], x_3),([x_3,x_4,x_5], x_6), ..., ([x_{T-3},x_{T-2},x_{T-1}], x_T)$ (Not using Rolling-windows). Using a batchsize of 1, the networkinput is $(1,3,n\_features)$, where $n\_features$ is the dimension of $X_i$ ($1$ for an univariat timeseries, $>1$ for multivariate timeseries).
Batches and timelags
So far so good. But, considering the batchsize and that within the batch the states of the Cells are not reset, the initial state of the second sample is the state at the end of the first sample. So with a batchsize of $2$ the network gets the input as shown in the picture below (Inputshape would be $(2, 3, n\_features)$).The number in the superscript indicates the sample. Doing so, we are messing up the timeseries, because, as you can see, the series now is $[x_0, x_1, x_2, x_1, x_2, x_3]$. (The same problem occurs between to batches, if the network is set to be stateful).
If the samples are not generated by rolling-windows, its obvious that there are missing values in the output sequence ($y_4$ and $y_6$ would be the outputs). And since there is no useful reason to do a seq-to-seq prediction for my problem (as stated previous), I'll ignore that case for now.
As there is a good reason to use batches to train RNNs (there is an article at machinelearningmastery) and not to update the weights after every sample, I am using a single timestep for each sample. The inputshape is now $(batch\_size, 1, n\_features)$. For example using a batchsize of 6 the networks calculation should look like this. (For stateful networks the next observation would be $x_6, ...$ together with the state at $x_5$, the batchsize determines when to update the weights during training.)
Truncated Backpropagation Through Time
As stated in this stackoverflow question the BPTT for keras RNNs is limited to the timelags of the input. Lets have a look at our input shape considering the BPTT.
For a higher order of lags, the error is back-propagated through time.
For the approach, where the input is
$(batch\_size, 1, n\_features)$ there is basically no BPTT, because it stops at one. And I think, that the weights of the recurrent connection are not updated at all, are they?
Conclusion and Question
Learning with a timelag of one might not be a good idea (mainly because of the insufficient BPTT, as the states can be preserved). Working with a higher order of timelags leads into a) holes in the output data using no rolling-windows while generating the samples or b) the time sequence is messed up.
What is a good approach to archive BPTT with Keras LSTMs, predicting future values (one-step and recursive multistep prediction as will) and still keep the input data stream-like (one observation after another)? Does some one have experienced similar problems and if so, how did you solve it?
I'm thankful for every input and maybe I also could help someone who is working on similar stuff ;)
My theoretical solution
While writing this question, I had this idea:
As far as i understand the Keras Layers, I can use TimeDistributed Outputlayer to have the Outputlayer applied to every given lag and the networks returns the sequence. (seq-to-seq network)
input = Input(batch_shape=(batch_size, timelags, features))
lstm = LSTM(units=10, return_sequences=True)(input)
out = TimeDistributed(Dense(units=features))(lstm)
To avoid messing up the sequence, I do not use a rolling-window. The Input/Outputs are (for lags=3) $([x_0,x_1,x_2], [x_1, x_2, x_3]),([x_3, x_4, x_4],[x_4, x_5, x_6])...)$ After I have trained the model I build a new one, which has the same architecture, but the inputshpape $(1, 1, features)$ and copy the trained parameters. This works, because the each LSTM- and outputpart as shown in the image are the actually the same (it shows the unrolled version of the lstm) and there is no need for BPTT while predicting, so no need to keep timelags > one and as the network is stateful the hiddenstates are preserved and the output should be the same as if I use a higher order of timelags with a sequence as output.
prediction_model.set_weights(trained_model.get_weights())
This way the BPTT is used and the input can be fed as stream of single points in time while predicting(I'm testing it now).
