When you build a classification or regression model, you typically split the data into a train data set and a test data set. The test data is a randomly selected subset of the overall data.

Once you are done with the training, you discard the test data, and apply the model you built to new unknown data.

But in the case of time series, this wouldn't work: You can't just randomly choose any subset of the data for your test data set, since the data is ordered. The test data set has to be composed of later observations of the data, while the train data set is made up of earlier observations of the data.

Say for example that you have data for 11 months of the year, and you want to forecast the values for December. You would train your model with the data from January through September, and then test it on the data from October and November.

But after you've successfully trained your model, what do you do with the test data?

On one hand, it doesn't make sense to forecast the values for December using a model built with Jan-Sep data. Such a model would miss any important trends that occurred in Oct and Nov.

On the other if you bring the data from Oct and Nov back into the model, the parameters of the model will change, and so we're no longer sure that we are going to get the same accuracy that we got when we trained the model with just the Jan-Sep data.

There seems to be a dilemma here.

So how is this problem approached when using machine learning models, especially non parametric models like Neural Networks for time series forecasting? Do they incorporate the test data into the model or do they discard it?

  • $\begingroup$ You could use all your data and assume the incorporation of the test set does not significantly affect performance, then verify this assumption when the actual data rolls in. As long as you have enough data to eliminate noise and capture seasonality you should be fine. If you don't then it is normal to miss "important trends". Based on the outcome you could refine your approach to use all the data if you don't think you have enough, and leave some out for validation if you think you do; i.e., increase the size of your hold out set over time, up to a point. $\endgroup$ – Emre Dec 12 '17 at 18:21

I agree with what Emre has commented on your question.

If you have enough data, I would try cross-validating your model by training it on different time sections of your data. For example, train your model on Jan-March data and then make predictions on April's data. Then you could train the model on Feb-April data, and predict May's data. Essentially creating a few training and testing sets to see how well your model generalizes. Once you feel you are getting good results, you could train your final "production" model on all of your available data and then keep an eye on how it does when real data comes in and make adjustments on what you see.

I've used a similar approach in the past on accelerometer data to classify exercises from a specific workout, and it worked well. Maybe something similar will work for you.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.