1
$\begingroup$

I have this time series below, that I divided into train, val and test: enter image description here Basically, I trained an ARIMA and an LSTM on those data, and results are completely different, in terms of prediction: ARIMA: enter image description here LSTM: enter image description here

Now, maybe I am passing, in some way, the test set to LSTM in order to perform better? Or LSTM is simply (lot) better than ARIMA? Below there is some code. Note that in order to do prediction in future days, I am adding the new and last predicted value to my series, before training and predicting: ARIMA code:

# Create list of x train valuess
history = [x for x in x_train]

# establish list for predictions
model_predictions = []

# Count number of test data points
N_test_observations = len(x_test)

# loop through every data point
for time_point in list(x_test.index[-N_test_observations:]):
    model = sm.tsa.arima.ARIMA(history, order=(3,1,3), seasonal_order=(0,0,0,7))
    model_fit = model.fit()
    output = model_fit.forecast()
    yhat = output[0]
    model_predictions.append(yhat)
    true_test_value = x_test[time_point]
    #history.append(true_test_value)
    history.append(yhat)
        
MAE_error = mean_absolute_error(x_test, model_predictions)
print('Testing Mean Squared Error is {}'.format(MAE_error))
Testing Mean Squared Error is 86.71141520892097

LSTM code:

def sequential_window_dataset(series, window_size):
    ds = tf.data.Dataset.from_tensor_slices(series)
    ds = ds.window(window_size + 1, shift=window_size, drop_remainder=True)
    ds = ds.flat_map(lambda window: window.batch(window_size + 1))
    ds = ds.map(lambda window: (window[:-1], window[1:]))
    return ds.batch(1).prefetch(1)

# reset any stored data
keras.backend.clear_session()
tf.random.set_seed(42)
np.random.seed(42)

# set window size and create input batch sequence
window_size = 30
train_set = sequential_window_dataset(normalized_x_train, window_size)
valid_set = sequential_window_dataset(normalized_x_valid, window_size)

# create model
model = keras.models.Sequential([
  keras.layers.LSTM(100, return_sequences=True, stateful=True,
                         batch_input_shape=[1, None, 1]),
  keras.layers.LSTM(100, return_sequences=True, stateful=True),
  keras.layers.Dense(1),
])

# set optimizer
optimizer = keras.optimizers.Nadam(lr=0.00033)

# compile model
model.compile(loss=keras.losses.Huber(),
              optimizer=optimizer,
              metrics=["mae"])

# reset states
reset_states = ResetStatesCallback()

#set up save best only checkpoint
model_checkpoint = keras.callbacks.ModelCheckpoint(
    "my_checkpoint", save_best_only=True)

early_stopping = keras.callbacks.EarlyStopping(patience=50)

# fit model
model.fit(train_set, epochs=500,
          validation_data=valid_set,
          callbacks=[early_stopping, model_checkpoint, reset_states])

# recall best model
model = keras.models.load_model("my_checkpoint")
# make predictions
rnn_forecast = model.predict(normalized_x_test[np.newaxis,:])
rnn_forecast = rnn_forecast.flatten()
# Example of how to iverse
rnn_unscaled_forecast = x_train_scaler.inverse_transform(rnn_forecast.reshape(-1,1)).flatten()
rnn_unscaled_forecast.shape

'LSTM': 9.964744041030935

Maybe there is something with that window size of the LSTM? Or maybe something when I do predictions for LSTM? # make predictions rnn_forecast = model.predict(normalized_x_test[np.newaxis,:])

$\endgroup$

1 Answer 1

3
$\begingroup$

Arima and LSTM are very different and there could be some tips to improve results.

Have you tried relative values instead of raw values?

For instance:

#Raw values: 
raw=[1200, 1300, 1250, 1370]

#Relative (or differential) values: 
diff=[+100,-50,+120]

Sometimes, raw values like 1400 could alter the results for ARIMA and LSTM differently.

On the other hand, LSTM could have bad predictions with noisy data. Some smoothing could improve results, but it depends on the kind of data.

Finally, are you trying to forecast 30 days in a single shot? Most predictions focus on 1-day forecast and set their precision on the sequential results from one day to another on the 30 days of validation data.

If your aim is to get accurate long-term forecasting, ARIMA and LSTM might not be the best solutions (overall ARIMA), because they have their own structural limitations. This could explain also why LSTM results have a gap with real results: some intern mechanisms have limited memory and wrongly predict important decreases or decreases of values.

The shape result of LSTM seems correct, but there is a small shift in Y of 10 because it initially predicted a smaller decrease. LSTM is quite difficult to understand: all I can say is that weights are connected to each other and peaks are more difficult to predict because of those dependencies. I recommend reading the initial paper, it's very interesting:

https://www.researchgate.net/publication/13853244_Long_Short-term_Memory

My advice is to lose accuracy by grouping values (ex: make a prediction of weeks instead of days) or use long-term models like those ones:

https://towardsdatascience.com/extreme-event-forecasting-with-lstm-autoencoders-297492485037

https://thuijskens.github.io/2016/08/03/time-series-forecasting/

https://arxiv.org/pdf/2210.08244.pdf

$\endgroup$
12
  • $\begingroup$ I have not understood the part of the relative values (+25, -32), can you explain me better? Then, yes, I am trying to forecast lot of days in the future, a long term forecasting. $\endgroup$
    – CasellaJr
    Oct 22, 2022 at 9:05
  • $\begingroup$ I've updated the answer with an example. If you are trying to predict many days, you should start with one and then increase the forecast. Otherwise, you may have poor results. Forecasting many days is in general very difficult. Indeed, the algorithm may not have enough memory to learn a lot of scenarios. $\endgroup$ Oct 23, 2022 at 7:41
  • 1
    $\begingroup$ I mean that your bad results could be merely explained by a too long forecast. $\endgroup$ Oct 24, 2022 at 7:58
  • 1
    $\begingroup$ I've explained more in detail in the answer. Hope it answers your question. $\endgroup$ Oct 24, 2022 at 13:24
  • 1
    $\begingroup$ I think you're right, but as I've never used ARIMA for long-term forecasting, I can't have an accurate point of view. My best advice is to try to simulate ARIMA forecast interactively with different horizon forecasts. machinelearningplus.com/time-series/… $\endgroup$ Oct 25, 2022 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.