LSTM Timeseries recursive prediction converge to same value

Question

I'm working on Timeseries sequence prediction using LSTM. My goal is to use window of 25 past values in order to generate a prediction for the next 25 values. I'm doing that recursively: I use 25 known values to predict the next value. Append that value as know value then shift the 25 values and predict the next one again until i have 25 new generated values (or more)

I'm using "Keras" to implement the RNN Architecture:

regressor = Sequential()
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.1))

regressor.add(Dense(units = 1))

regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')

regressor.fit(X_train, y_train, epochs = 10, batch_size = 32)

Problem: Recursive prediction always converge to the some value no matter what sequence comes before.

For sure this is not what I want, I was expecting that the generated sequence will be different depending on what I have before and I'm wondering if someone have an idea about this behavior and how to avoid it. Maybe I'm doing something wrong ...

I tried different epochs number and didn't help much, actually more epochs made it worse. Changing Batch Size, Number of Units , Number of Layers , and window size didn't help too in avoiding this issue.

I'm using MinMaxScaler for the data.

Answer 1

If you want to use an LSTM for time series prediction, you have to try every possible parameter to find the best possible result. In order to check your model performance, you should also check your train/validation loss (this will probably show that 10 epochs is too few). Check your network architecture. Why do you use three layers? More layers won't mean better results. Try different optimizers with different learning rates. Finally, ask yourself why you are even using an LSTM for prediction? Have you tried other methods? LSTMs aren't magic, they won't have perfect prediction without any pattern to draw from, and you have pretty small samples for training.

Answer 2

It seems as if your RNN is far too complex for modeling a univariate time series: I would advise using a simpler model, using fewer LSTM layers.

Now, time series forecasting always has this problem of converging to some mean value in the long term. Real time series are believed to be subjected to 'shocks' from the 'environment' that result in sudden increase or decrease. If you do not have these shocks in your data, and make no assumption about when they occur, the only sensible long-term forecast would be some constant value. Do you find this intuitive?

Secondly, using an LSTM for recursive forecasting the way you do leads to an very significant accumulation of errors. For example, when predicting for time step t + 3, both the hidden states of the model and the data that you input (input in this case: your prediction for t + 2) are subjected to some error. During training, only the hidden states were subjected to error: the input were actual observations. Do you see how this could lead to this extreme forecast runaway?

Answer 3

For regression, typically the activation function stays linear