2
$\begingroup$

I am training an Xgboost using 60% of my data and use 40% for testing.

In the 60% of data, I use 5-fold validation to find the best number of trees. I find that the optimal number of trees is around 150.

I evaluate my model in the 40% of the data that I have left and I am happy with the performance, so now I can deploy my model. However, before that, I want to do a final retrain to make use of all my data. I wonder if the number of trees obtained in cross-validation is going to be optimal when I train with the full dataset as well. My intuition says that I should use more than 150 trees as I have more data and I can overfit less.

Are there any sound decisions regarding the number of trees of the retrained model?

In my mind I have at least 3:

  • Use the full data to cross-validate and obtain the optimal number of trees (this is ok for sure but it is the slowest).
  • Use the same number of trees (this is the most conservative, we might be underfitting the data).
  • Use a heuristic, like final_trees = 150*100/60. I am very interested in a legit heuristic where I would not underfit and where I don't need to train models on cross-validation again.

Have you heard of any heuristic like that?

Note: this is not only specific to Xgboost, as any model with a parameter that controls regularization can also have the same issues.

$\endgroup$
  • $\begingroup$ I agree with Noah's answer. Let me ask: What benefit do you expect from re-training the model? It should have learned what it can learn (and what can be tested!!). So re-training the model will introduce a hefty risk of underperformance in final production, as I see it. Eventually, there is a lot to lose and little to gain in my view. $\endgroup$ – Peter Dec 25 '19 at 12:06
  • $\begingroup$ Two reasons I can think of: 1) In a product setting, you might be using your most recent data to test. These data are the most relevant and it makes sense to train your final model on these data. 2) You might be training with very small portion of the data to iterate and work fast. However, you want your final model to be more performant thus you want more data to train. $\endgroup$ – David Masip Dec 27 '19 at 11:01
  • $\begingroup$ And I see your point, what can be tested is already learnt, I think this is pretty interesting. $\endgroup$ – David Masip Dec 27 '19 at 11:05
  • $\begingroup$ I think if you would use robust methods such as OLS, you would have no problem. However, boosting can be quite sensitive, so that there is a real risk of making things much worse by re-training without proper testing. I would not do this to be honest. $\endgroup$ – Peter Dec 27 '19 at 11:19
3
+50
$\begingroup$

Unpopular opinion: Second quickest way to overfit (next to data-leakage) is hyper-parameter optimization.

Why? You are assuming you wont have covariate-shift, while in most of the cases you can bet on it. Hence optimising too much on train (available data) will be ruin.

The most reasonable assumption (that we have to make sure it stands) is that 60% of the data is representative of the 40% and also approximately of the future unseen data hence all of the 150 trees should catch all of the neccesarry information.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You perform CV to get insights into how model will behave in prod environment. Retraining model on whole data could possibly change your domain and you won't see it in metrics. That's why I'd advise strongly against that. With splits, you at least have some sense what's going to production. Your objective is to train a model that generalizes well. Training and tuning model on whole sample is just a highway to overfit hell. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 17 '19 at 13:36
  • $\begingroup$ @PiotrRarus, If I well understood OP idea, training on the whole dataset is not just one training, but another 5fold training using 100% of data instead of 60%, so both the validation metric will be visible and the advantage of bagging 5 models will be kept. However, I am skeptical regarding concentration on the number of trees. $\endgroup$ – Grzegorz Sionkowski Dec 22 '19 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.