# LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process.

def my_process(n, p, drift=0, displacement=0):
x = np.zeros(n)

for i in range(1, n):
x[i] = drift * i + p * x[i-1] + (1-p) * np.random.randn()
return x + displacement


Then I built an LSTM model in Keras, following this example. I simulated processes with high autocorrelation p=0.99 of length n=10000, trained the neural network on the first 80% of it and let it do one-step-ahead predictions for the remaning 20%.

If I set drift=0, displacement=0, everything works fine:

Then I set drift=0, displacement=10 and things went pear-shaped (notice the different scale on the y-axis):

This is not terribly surprising: LSTMs should be fed with normalized data! So I normalized the data by rescaling it to the interval $$[-1, 1]$$. Phew, things are fine again:

Then I set drift=0.00001, displacement=10, normalized the data again and ran the algorithm on it. This does not look good:

Apparently the LSTM cannot deal with a drift. What to do? (Yes, in this toy example I could simply subtract the drift; but for real-world time series, this is much harder). Maybe I could run my LSTM on the difference $$X_{t} - X_{t-1}$$ instead of the original time series $$X_t$$. This will remove any constant drift from the time series. But running the LSTM on the differenced time series does not work at all:

My question: Why does my algorithm break down when I use it on the differenced time series? What is a good way to deal with drifts in time series?

Here is the full code for my model:

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM
from keras.models import Sequential

# The LSTM model
my_model = Sequential()

my_model.compile(loss='mse', optimizer='rmsprop')

def my_prediction(x, model, normalize=False, difference=False):
# Plot the process x
plt.figure(figsize=(15, 7))
plt.subplot(121)
plt.plot(x)
plt.title('Original data')

n = len(x)
thrs = int(0.8 * n)    # Train-test split
# Save starting values for test set to reverse differencing
x_test_0 = x[thrs + 1]
# Save minimum and maximum on test set to reverse normalization
x_min = min(x[:thrs])
x_max = max(x[:thrs])

if difference:
x = np.diff(x)   # Take difference to remove drift
if normalize:
x = (2*x - x_min - x_max) / (x_max - x_min)   # Normalize to [-1, 1]

# Split into train and test set. The model will be trained on one-step-ahead predictions.
x_train, y_train, x_test, y_test = x[0:(thrs-1)], x[1:thrs], x[thrs:(n-1)], x[(thrs+1):n]

x_train, x_test = x_train.reshape(-1, 1, 1), x_test.reshape(-1, 1, 1)
y_train, y_test = y_train.reshape(-1, 1), y_test.reshape(-1, 1)

# Fit the model
model.fit(x_train, y_train, batch_size=200, epochs=10, validation_split=0.05, verbose=0)

# Predict the test set
y_pred = model.predict(x_test)

# Reverse differencing and normalization
if normalize:
y_pred = ((x_max - x_min) * y_pred + x_max + x_min) / 2
y_test = ((x_max - x_min) * y_test + x_max + x_min) / 2
if difference:
y_pred = x_test_0 + np.cumsum(y_pred)
y_test = x_test_0 + np.cumsum(y_test)

# Plot estimation
plt.subplot(122)
plt.plot(y_test[-100:], label='Actual data')
plt.title('Prediction on test set')
plt.legend()
plt.show()

# Make plots
x = my_process(10000, 0.99, drift=0, displacement=0)
my_prediction(x, my_model, normalize=False, difference=False)

x = my_process(10000, 0.99, drift=0, displacement=10)
my_prediction(x, my_model, normalize=False, difference=False)

x = my_process(10000, 0.99, drift=0, displacement=10)
my_prediction(x, my_model, normalize=True, difference=False)

x = my_process(10000, 0.99, drift=0.00001, displacement=10)
my_prediction(x, my_model, normalize=True, difference=False)

x = my_process(10000, 0.99, drift=0.00001, displacement=10)
my_prediction(x, my_model, normalize=True, difference=True)


Looking again at your autocorrelated process:

    def my_process(n, p, drift=0, displacement=0):
x = np.zeros(n)

for i in range(1, n):
x[i] = drift * i + p * x[i-1] + (1-p) * np.random.randn()
return x + displacement


It looks like things are breaking down when the value of displacement is high. This makes sense, as you say, because LSTMs need normalized data.

The drift parameter is a bit different. When a small amount of drift is included, since p is large, the amount of drift is similar to the amount of random noise being added via np.random.randn().

In the plots for drift=0.00001, displacement=10, it looks like the predictions would be fine except for the y-shift. Because of this, I think the root of the problem is still in the displacement parameter, not the drift parameter. Differencing, as has been done, will not help with the displacement parameter; instead, it corrects for drift.

I can't tell from your code, but it looks like perhaps the displacement was not accounted for in model.predict. That's my best guess.

• Thank you for looking at it! Differencing will help with the displacement parameter though: $(X_{t+1} + c) - (X_t + c) = X_{t+1} - X_t$. Also, the last example uses normalization (after differencing), so that should not be an issue... Mar 3 '18 at 2:18
• Hi again, okay, good point! Hmm. I think what you call 'drift', I would call a moving average (I hope). You could try including some kind of covariate into your model to account for the moving average. Ideally an LSTM would discover that on its own, of course, but here it seems to be stuck. Mar 4 '18 at 15:04
• I am a bit concerned about this because I want to apply LSTMs to stock prices. They do have a drift/moving average, and the standard approach (in statistics, at least) is to apply differencing, i.e. use returns instead of prices. So I would like to understand why this does not seem to work (even for such a simple model) with LSTMs. Mar 5 '18 at 11:01
• Have you traced through the forward and backward passes with your original values vs differenced values? I wonder if there's some kind of vanishing gradient problem happening with the differenced values. LSTMs are of course more robust to this, but they can run into these kinds of problems, so it might be worth a look. Mar 6 '18 at 15:59

When you choose x_min and x_max, you are choosing it from 1:threshold alone. Since your series is monotonically increasing (well almost..), the testing values are all values > 1. This, the LSTM model has never seen during training at all.

Is that why you are seeing what you are seeing?

Can you try the same with x_min and x_max coming from the whole dataset instead?

• This might work in my toy example; but if I use LSTMs to actually predict something, this would require looking into the future. Jul 23 '18 at 12:59