How to choose validation set for production environment?

Question

I am using XGBoost for a time-series regression problem.

During development, i choose my validation set on last %10 percentage of data. Using timeseries split cross validation and grid-search, I got my best model on this with corresponding xgb hyperparameters.

My question is, how to choose validation set (for early stopping) on my production environment?

1) i have chosen last %10 percentage of my data as validation set, but this set is also included on training data. therefore overfit. very sensitive to noisy data.

2) my predicted data (lets say Y) changes over time, when i choose random rows within last year (%10 amount) and dont include them in training set, it gave me worse results on production than first option.

3) when i choose last week's data as validation data, not included in training set, it gave better result on 2. option. but i am not including last week's data to training procedure.

4) Or do i need a validation set on my production environment? should i set iteration count from the experiments done in development stage? (e.g i got best result on 10k th iteration, so i should limit my production-setup iteration count with 10k without using validation set at all?)

--

So, how can i choose validation set for my production environment? Best practices, or are there any tips/tricks for this?

Answer 1

If you're simply re-training the XGBoost model periodically in order to account for the changing nature of your data, the best option is to hold out a recent set of data for testing (some variation of your option 3).

As you mention, option 1- that is, training on the entire available set and validating on the most recent 10 percent of the data, is very likely to overfit (and therefore overestimate its performance on future data).

Option 2 has the downfall that you're attempting to predict information that you have post-hoc (after the fact) information about- that is, you know what subsequent observation values were, which is very relevant to a time-series prediction. For example, if you're trying to predict what the value is tomorrow, and you know what the value is for the day after tomorrow already, you can do a much better job of predicting tomorrow. Unfortunately, you can never know this before the day after tomorrow begins.

Therefore, option 3 is the most valuable practice to determine the likelihood of your model to accurately predict future observations- you're using only data that would be available from the point of prediction (in the past relative to the predicted period) and holding that data out of the training set for the model.

The best practice in these circumstances is to hold yourself accountable for "not cheating"- that is, try your best to provide a fair and honest experiment on the prediction of your model on data it hasn't seen and would be available in the context of a future use situation.