How to optimally train deep learning model using output as new input

Question

I'm trying to train a network to predict the future. My current setup uses 5 time steps as inputs from the past, each consisting of 10 features, resulting in a [5, 10] input matrix (initially generated by another algorithm). Each of these time steps is a dt step further in time. The network outputs 1 output with again 10 features, a dt step in the future w.r.t the last time step from the inputs. This input replaces the oldest time step in the inputs matrix.

The network is trained using a generated input sequence as well as one correct output for that sequence. The MSE error reduces to about 1e-4 for a 100.000 samples training set.

When executing the network using the previously mentioned method. The solution blows up pretty quickly as the network makes an error in the first prediction and uses that as an input, removing a correct but older input. Now, a bigger error is made due a partially incorrect input sequence which it has never seen before. The next time step this becomes even worse and so on.

Instead of training with a completely correct input sequence and its correct output, i was thinking the network should train on its own predicted, not completely correct, inputs as well. I could only think of training the network as mentioned before. Use that to create more samples including its own prediction errors and train it again using that training set. That process could be repeated until a desired point.

Is there a more efficient way to do that training only on a single training set using Keras as my deep learning toolbox?

Edit

I will provide some more context for my problem. I'm trying to predict the future states for a few types (square, triangle and sawtooth) of travelling waves, moving with a certain speed to the left or right. The following figure gives an example of a random travelling wave I could be using during training.

These waves are discretized by 1024 x positions with a y value. Due to their relative high dimensionality, I use an autoencoder to create a latent space with a dimensionality of 10. The following figure gives an example of an autoencoded wave, including the input, encoded latent space and the decoded output.

The lower dimensional latent space can be advanced in time more easily for an RNN network. To do this, I create five time steps of the wave using a generation algorithm. These are encoded to their latent space using the encoder of the autoencoder. Now they are fed into an RNN which predicts the next time step of the latent space. This output is fed back to the inputs to replace the oldest time step input. The output can also be decoded using the decoder of the autoencoder, resulting in a high dimensional prediction of the next time step of the traveling wave.

I have trained the RNN network giving it examples of 5 latent inputs and the correct next time step latent space output, reaching an mse of about 1.5e-4. A result for a new prediction of this RNN can be seen in the following figure.

The first prediction are pretty accurate for most of the inputs I give the network. However, when I use its own outputs as new inputs to keep on predicting next time steps, the error increases. The next figure gives an example of the same wave depicted before but now at prediction step 20.

For the RNN i'm currently using the following Keras setup:

inputLayer = Input(shape = (RNN_numInputs, AE_latentSize))
x = CuDNNLSTM(RNN_units[0], return_sequences=False)(inputLayer)
x = Dropout(0.2)(x)
x = RepeatVector(RNN_numOutputs)(x)  
x = CuDNNLSTM(RNN_units[1], return_sequences=True)(x) 
x = Dropout(0.2)(x)
x = Conv1D(AE_latentSize, RNN_outputKernelSize, activation='linear', padding = 'same')(x)

rnn = Model(inputLayer, x)

Where RNN_numInputs is 5, AE_latentSize is 10, RNN_units is [200, 400] and RNN_outputKernelSize is 3.

I think that when I can decrease the error of the prediction even further, the long term prediction results can become better.

Does anyone have an idea how I can increase the perfomance of this algorithm?

Answer 1

The first point I would make (and you are likely aware of this given you specified "lstm" as part of your tags), is that you should use a long-short term memory network for this purpose.

Given that you are working with time series data, the risk with a regular neural network is that it treats all the observations as independent when this is not the case with a time series.

Instead of training with a completely correct input sequence and its correct output, i was thinking the network should train on its own predicted, not completely correct, inputs as well.

This is somewhat of a slippery slope, because one must assume that you are using actual outputs. The model might be falsely trained to believe that the predicted values (which ordinarily would not yield the correct output) do in this case, and this skews your results. The model is only as good as the data that is being fed into it - if the data is wrong then the results will be as well.

In general, your error reading looks quite good, although I am not familiar with your data or model.

If you wanted to generate dummy data, one possibility could be to use a bootstrap simulation of sorts. For instance, suppose you have a time series with 10% mean and 20% volatility. You then choose to generate a random time series based on these two readings. The neural network could then be further trained on such data, and may potentially improve accuracy.

However, I would be cautious of mixing inaccurate inputs with correct outputs - as this has the potential to skew your results.