Why so discrepancy between ARIMA and LSTM in time series forecasting?

Question

I have this time series below, that I divided into train, val and test: Basically, I trained an ARIMA and an LSTM on those data, and results are completely different, in terms of prediction: ARIMA: LSTM:

Now, maybe I am passing, in some way, the test set to LSTM in order to perform better? Or LSTM is simply (lot) better than ARIMA? Below there is some code. Note that in order to do prediction in future days, I am adding the new and last predicted value to my series, before training and predicting: ARIMA code:

# Create list of x train valuess
history = [x for x in x_train]

# establish list for predictions
model_predictions = []

# Count number of test data points
N_test_observations = len(x_test)

# loop through every data point
for time_point in list(x_test.index[-N_test_observations:]):
    model = sm.tsa.arima.ARIMA(history, order=(3,1,3), seasonal_order=(0,0,0,7))
    model_fit = model.fit()
    output = model_fit.forecast()
    yhat = output[0]
    model_predictions.append(yhat)
    true_test_value = x_test[time_point]
    #history.append(true_test_value)
    history.append(yhat)
        
MAE_error = mean_absolute_error(x_test, model_predictions)
print('Testing Mean Squared Error is {}'.format(MAE_error))
Testing Mean Squared Error is 86.71141520892097

LSTM code:

def sequential_window_dataset(series, window_size):
    ds = tf.data.Dataset.from_tensor_slices(series)
    ds = ds.window(window_size + 1, shift=window_size, drop_remainder=True)
    ds = ds.flat_map(lambda window: window.batch(window_size + 1))
    ds = ds.map(lambda window: (window[:-1], window[1:]))
    return ds.batch(1).prefetch(1)

# reset any stored data
keras.backend.clear_session()
tf.random.set_seed(42)
np.random.seed(42)

# set window size and create input batch sequence
window_size = 30
train_set = sequential_window_dataset(normalized_x_train, window_size)
valid_set = sequential_window_dataset(normalized_x_valid, window_size)

# create model
model = keras.models.Sequential([
  keras.layers.LSTM(100, return_sequences=True, stateful=True,
                         batch_input_shape=[1, None, 1]),
  keras.layers.LSTM(100, return_sequences=True, stateful=True),
  keras.layers.Dense(1),
])

# set optimizer
optimizer = keras.optimizers.Nadam(lr=0.00033)

# compile model
model.compile(loss=keras.losses.Huber(),
              optimizer=optimizer,
              metrics=["mae"])

# reset states
reset_states = ResetStatesCallback()

#set up save best only checkpoint
model_checkpoint = keras.callbacks.ModelCheckpoint(
    "my_checkpoint", save_best_only=True)

early_stopping = keras.callbacks.EarlyStopping(patience=50)

# fit model
model.fit(train_set, epochs=500,
          validation_data=valid_set,
          callbacks=[early_stopping, model_checkpoint, reset_states])

# recall best model
model = keras.models.load_model("my_checkpoint")
# make predictions
rnn_forecast = model.predict(normalized_x_test[np.newaxis,:])
rnn_forecast = rnn_forecast.flatten()
# Example of how to iverse
rnn_unscaled_forecast = x_train_scaler.inverse_transform(rnn_forecast.reshape(-1,1)).flatten()
rnn_unscaled_forecast.shape

'LSTM': 9.964744041030935

Maybe there is something with that window size of the LSTM? Or maybe something when I do predictions for LSTM? # make predictions rnn_forecast = model.predict(normalized_x_test[np.newaxis,:])

Answer 1

Arima and LSTM are very different and there could be some tips to improve results.

Have you tried relative values instead of raw values?

For instance:

#Raw values: 
raw=[1200, 1300, 1250, 1370]

#Relative (or differential) values: 
diff=[+100,-50,+120]

Sometimes, raw values like 1400 could alter the results for ARIMA and LSTM differently.

On the other hand, LSTM could have bad predictions with noisy data. Some smoothing could improve results, but it depends on the kind of data.

Finally, are you trying to forecast 30 days in a single shot? Most predictions focus on 1-day forecast and set their precision on the sequential results from one day to another on the 30 days of validation data.

If your aim is to get accurate long-term forecasting, ARIMA and LSTM might not be the best solutions (overall ARIMA), because they have their own structural limitations. This could explain also why LSTM results have a gap with real results: some intern mechanisms have limited memory and wrongly predict important decreases or decreases of values.

The shape result of LSTM seems correct, but there is a small shift in Y of 10 because it initially predicted a smaller decrease. LSTM is quite difficult to understand: all I can say is that weights are connected to each other and peaks are more difficult to predict because of those dependencies. I recommend reading the initial paper, it's very interesting:

https://www.researchgate.net/publication/13853244_Long_Short-term_Memory

My advice is to lose accuracy by grouping values (ex: make a prediction of weeks instead of days) or use long-term models like those ones:

https://towardsdatascience.com/extreme-event-forecasting-with-lstm-autoencoders-297492485037

https://thuijskens.github.io/2016/08/03/time-series-forecasting/

https://arxiv.org/pdf/2210.08244.pdf