How to compare different forecasting models over different time horizon?

Developed multiple Models with AR, ARIMA, VAR; LSTM , SARIMA. Now, the purpose is to find out which model performs best on a given use case with different time horizons.

The time series data is aggregated in weekly time-stamps.

A train-test split of 80/20 percent is performed. The question is: Which model is best for a 1 month, 6 months and 12 Months forecast?

How to evaluate those with RMSE? Should the testset size change everytime? For example, use the 80% training data to train and predict on testdata[:10], testdata[:20], testdata[:30] and so on. Is there an other methology?

A simple AR Model and it seems like it is doing worse with a testdata[:10] prediction than with the whole testdata set.

• Fixing the training set for all models and changing the tail size (test data) would be a good approach. However, repeatedly doing this with OOS rolling validation would give a better estimate of the performance of different models. Sep 22 '21 at 19:25
• Hey what do you mean by rOOS? But is it normal that the Forecast for 1 Time Stamp or for 10 of the AR Model is worse than the whole test data set which would be about 1 year? Sep 22 '21 at 21:13
• rOOS, I was referring to technique introduced by Hydman, here robjhyndman.com/hyndsight/tscv . One would need to estimate with rOOS if AR model behaves like that. Sep 22 '21 at 21:49
• @MehmetSüzen i see. But is this a scientific war of Comparing Models ? Or should i use 90% Training data for 10% testdata Evaluation and 80 for 20 and so on? Or is or okay to use 80% for arima lstm and AR and zehn only Manipulate the testdata[0:forecasthorizon] Sep 22 '21 at 22:01