Time series test data dilema

Question

I’m trying to build a model to predict the amount of sales of a product for the next few days

This question is about whether or not I should use the tail of the serie as the test set and train models using the rest of the data or I should create a test set picking dates at random as usual

Reading about classical time series models (ARIMA), they recommend the first approach (using the last days as test) but I feel weird doing this when applying a machine learning model

What is the correct approach? Any advantage or disadvantage using one or the other?

Answer 1

You can still do cross-validation with time series, but don't just take data points at random. A rolling window is a good way, like:

1st: ~~~~ train ~~~~ train ~~~~ test
2nd: ~~~~ train ~~~~ train ~~~~ train ~~~~ test
3rd: ~~~~ train ~~~~ train ~~~~ train ~~~~ train ~~~~ test

Where test size is about the same size as your forecast horizon

Answer 2

You could also split your data like this:

~~~~ train ~~~~ test ~~~~ train ~~~~ test ~~~~ ...

Then, you always use one pair of train and test to train and test the model. you can then tune your model hyperparameters based on the average loss it achieves on all the individual test sets in individual runs, given the current value of the hyperparameter. make sure you do not feed any of the test data (also not the lookback window) into the training procedure. this is crucial. compared to the other suggested approach, mine has the advantage that the training set size is always the same. his has the advantage that you train the model on more data.

Answer 3

You should not use shuffle for time series data as it will introduce lookahead bias. You can refer to below video as an example of how to do train test split for time series data. Tail should be test set, there are approaches to do cross validation too, but tail should always be test set in each iteration.

Link: https://www.youtube.com/watch?v=vV12dGe_Fho