I'm trying to train a network to predict the future. My current setup uses 5 time steps as inputs from the past, each consisting of 10 features, resulting in a [5, 10] input matrix (initially generated by another algorithm). Each of these time steps is a dt step further in time. The network outputs 1 output with again 10 features, a dt step in the future w.r.t the last time step from the inputs. This input replaces the oldest time step in the inputs matrix.
The network is trained using a generated input sequence as well as one correct output for that sequence. The MSE error reduces to about 1e-4 for a 100.000 samples training set.
When executing the network using the previously mentioned method. The solution blows up pretty quickly as the network makes an error in the first prediction and uses that as an input, removing a correct but older input. Now, a bigger error is made due a partially incorrect input sequence which it has never seen before. The next time step this becomes even worse and so on.
Instead of training with a completely correct input sequence and its correct output, i was thinking the network should train on its own predicted, not completely correct, inputs as well. I could only think of training the network as mentioned before. Use that to create more samples including its own prediction errors and train it again using that training set. That process could be repeated until a desired point.
Is there a more efficient way to do that training only on a single training set using Keras as my deep learning toolbox?
I will provide some more context for my problem. I'm trying to predict the future states for a few types (square, triangle and sawtooth) of travelling waves, moving with a certain speed to the left or right. The following figure gives an example of a random travelling wave I could be using during training.
These waves are discretized by 1024 x positions with a y value. Due to their relative high dimensionality, I use an autoencoder to create a latent space with a dimensionality of 10. The following figure gives an example of an autoencoded wave, including the input, encoded latent space and the decoded output.
The lower dimensional latent space can be advanced in time more easily for an RNN network. To do this, I create five time steps of the wave using a generation algorithm. These are encoded to their latent space using the encoder of the autoencoder. Now they are fed into an RNN which predicts the next time step of the latent space. This output is fed back to the inputs to replace the oldest time step input. The output can also be decoded using the decoder of the autoencoder, resulting in a high dimensional prediction of the next time step of the traveling wave.
I have trained the RNN network giving it examples of 5 latent inputs and the correct next time step latent space output, reaching an mse of about 1.5e-4. A result for a new prediction of this RNN can be seen in the following figure.
The first prediction are pretty accurate for most of the inputs I give the network. However, when I use its own outputs as new inputs to keep on predicting next time steps, the error increases. The next figure gives an example of the same wave depicted before but now at prediction step 20.
For the RNN i'm currently using the following Keras setup:
inputLayer = Input(shape = (RNN_numInputs, AE_latentSize)) x = CuDNNLSTM(RNN_units, return_sequences=False)(inputLayer) x = Dropout(0.2)(x) x = RepeatVector(RNN_numOutputs)(x) x = CuDNNLSTM(RNN_units, return_sequences=True)(x) x = Dropout(0.2)(x) x = Conv1D(AE_latentSize, RNN_outputKernelSize, activation='linear', padding = 'same')(x) rnn = Model(inputLayer, x)
Where RNN_numInputs is 5, AE_latentSize is 10, RNN_units is [200, 400] and RNN_outputKernelSize is 3.
I think that when I can decrease the error of the prediction even further, the long term prediction results can become better.
Does anyone have an idea how I can increase the perfomance of this algorithm?