# Time series forecasting with RNN(stateful LSTM) produces constant values

I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values.

I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

Is this the correct way to forecast time series with LSTMs?

How can this behaviour be explained?

Will this be the case even for a time series with great seasonality?

Is this behaviour expected even for a very large time series data?

Should I rather try to train the network with Y matrix having 31st day to 60th day values for the same X?

My train data looks something like this:

array([[-0.35811423, -0.22393472, -0.39437897, ..., -0.36718042, -0.37080689, -0.35267452], [-0.22393472, -0.39437897, -0.13327289, ..., -0.37080689, -0.35267452, -0.2030825 ], [-0.39437897, -0.13327289, -0.1532185 , ..., -0.35267452, -0.2030825 , -0.25294651],

Architecture: Input LSTM layer (20 neurons) 1 hidden lstm layer 20 neurons 1 output dense layer, batch size as 1. Stateful lstm, I reset model states after each epoch.

• What library are you using? Usually, yes you can do exactly what you mentioned in terms of step by step forecast, as opposed to 30 day forecasts. – Landmaster Aug 8 '17 at 4:05
• Keras. But am getting constants with the step by step forecast. – Narahari B M Aug 8 '17 at 9:01
• Do you mind sharing your code and data on GitHub or somewhere else? – Louis T Nov 8 '17 at 1:28

### Is this the correct way to forecast time series with LSTMs?

Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values

You method could work in theory. But typically, for vanilla LSTM, the setup looks like $x = [x_0, x_1, ..., x_{n-1}]$ and $y = [x_1, x_2 ..., x_n]$. I suspect you will get much better result if you do it this way.

I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

This is not recommended because any error in prediction is compounded over time. please find my answer in the following post for a detailed explanation Is time series multi-step ahead forecasting a sequence to sequence problem?

### How can this behaviour be explained?

If you are getting constant prediction value, the chance is you are having so-called "gradient explosion problem", try to clip the gradient before backprop might be the solution.

Correction. Had a bit better thinking about it. it is not a gradient explosion problem here. More likely it is due to saturated neuron. Because LSTM uses sigmoid function, if the input is not properly scaled or weight is initialised inproperly, it could cause neuron saturation very quickly.

• Thanks Louis, But LSTMs dont suffer from Gradient Explosion or Vanishing Gradient problem right? – Narahari B M Nov 8 '17 at 8:48
• had a bit thinking about it. I think you are right. It's not the Gradient Explosion causing the problem. It should be saturated neuron instead. I edited my answer accordingly – Louis T Nov 8 '17 at 12:15

I am working on data for 3 years(daily data) and can see the values changing for all the days. I am trying to predict for next 30 days using Dr Jason Brownlee's tutorial http://machinelearningmastery.com/multi-step-time-series-forecasting-long-short-term-memory-networks-python/
The above tutorial helped me predict for next 30 days, I am still trying to improve on the predictions .

EDIT :While trying to improve my results I was experimenting and came across similar situation as yours. Recommendation --->Remove any activation functions from your network and it may help.

• I dont suppose you can remove activation functions, if you dont provide any activation function, then in keras the default activation function is linear. – Narahari B M Aug 11 '17 at 17:01
• exactly . that's what i am asking you to try. I just tried and it's giving nice predictions to me that i have validated (as your model may have linear relations) – Nitin Mahajan Aug 11 '17 at 17:04
• Thanks for this,I have always used linear. But what am asking is using a predicted value to predict the next values, this is when am getting constants, which is not what the tutorial you have followed is doing. Forecasting directly for arbitrary number of days works well except when you have to build a solution where in forecasting period can vary. Besides it will omit values at the end of the series, which is something i can't always afford. – Narahari B M Aug 11 '17 at 17:27