When forecasting time series, how does one incorporate the test data back into the model after training?

Question

When you build a classification or regression model, you typically split the data into a train data set and a test data set. The test data is a randomly selected subset of the overall data.

Once you are done with the training, you discard the test data, and apply the model you built to new unknown data.

But in the case of time series, this wouldn't work: You can't just randomly choose any subset of the data for your test data set, since the data is ordered. The test data set has to be composed of later observations of the data, while the train data set is made up of earlier observations of the data.

Say for example that you have data for 11 months of the year, and you want to forecast the values for December. You would train your model with the data from January through September, and then test it on the data from October and November.

But after you've successfully trained your model, what do you do with the test data?

On one hand, it doesn't make sense to forecast the values for December using a model built with Jan-Sep data. Such a model would miss any important trends that occurred in Oct and Nov.

On the other if you bring the data from Oct and Nov back into the model, the parameters of the model will change, and so we're no longer sure that we are going to get the same accuracy that we got when we trained the model with just the Jan-Sep data.

There seems to be a dilemma here.

So how is this problem approached when using machine learning models, especially non parametric models like Neural Networks for time series forecasting? Do they incorporate the test data into the model or do they discard it?

Answer 1

I agree with what Emre has commented on your question.

If you have enough data, I would try cross-validating your model by training it on different time sections of your data. For example, train your model on Jan-March data and then make predictions on April's data. Then you could train the model on Feb-April data, and predict May's data. Essentially creating a few training and testing sets to see how well your model generalizes. Once you feel you are getting good results, you could train your final "production" model on all of your available data and then keep an eye on how it does when real data comes in and make adjustments on what you see.

I've used a similar approach in the past on accelerometer data to classify exercises from a specific workout, and it worked well. Maybe something similar will work for you.

Answer 2

For future readers, the way I approached this was to write a testing iterator/evaluator which takes an existing dataset and chops it at a certain moment into two parts for training and testing. Then a new model is trained on the training set, and its forecasts are evaluated against the future test set. The whole process is repeated, inching the cut moment forward by some increment, and discarding the model.

I varied two things: a) the amount of test data because I wanted to learn how much historical data was needed for the model to settle, and b) the moment of cut to simulate how the system will work in the real world at any moment in time.

Here are some example training runs.

The blue-red line is the real data. The blue part is training data which the model has seen. The red part is test data which is kept hidden from the model, and so can be used to evaluate the forecast against reality. The yellow line is the model's forecast of the future, always starting from the very end of the training data. The performance is measured by some cross correlation technique between yellow and red.

BTW I'm not sure I understand what you mean about incorporating the test data into the training. Normally you'd never do that, else it skews the results to seem more accurate. In fact you're not even supposed to do parametric training from the test data results, because this indirectly injects attributes back into the model which should remain unseen, hence the use of an intermediate "evaluation" set to test each cycle (which strictly speaking is more training data), and the test data is only used for final evaluation after the parameters are fixed.