Developed multiple Models with AR, ARIMA, VAR; LSTM , SARIMA. Now, the purpose is to find out which model performs best on a given use case with different time horizons.

The time series data is aggregated in weekly time-stamps.

A train-test split of 80/20 percent is performed. The question is: Which model is best for a 1 month, 6 months and 12 Months forecast?

How to evaluate those with RMSE? Should the testset size change everytime? For example, use the 80% training data to train and predict on testdata[:10], testdata[:20], testdata[:30] and so on. Is there an other methology?

A simple AR Model and it seems like it is doing worse with a testdata[:10] prediction than with the whole testdata set.

  • $\begingroup$ Fixing the training set for all models and changing the tail size (test data) would be a good approach. However, repeatedly doing this with OOS rolling validation would give a better estimate of the performance of different models. $\endgroup$ Sep 22 '21 at 19:25
  • $\begingroup$ Hey what do you mean by rOOS? But is it normal that the Forecast for 1 Time Stamp or for 10 of the AR Model is worse than the whole test data set which would be about 1 year? $\endgroup$ Sep 22 '21 at 21:13
  • $\begingroup$ rOOS, I was referring to technique introduced by Hydman, here robjhyndman.com/hyndsight/tscv . One would need to estimate with rOOS if AR model behaves like that. $\endgroup$ Sep 22 '21 at 21:49
  • $\begingroup$ @MehmetSüzen i see. But is this a scientific war of Comparing Models ? Or should i use 90% Training data for 10% testdata Evaluation and 80 for 20 and so on? Or is or okay to use 80% for arima lstm and AR and zehn only Manipulate the testdata[0:forecasthorizon] $\endgroup$ Sep 22 '21 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.