I have a simple Question:

I am using XGBoost to classify some data: 1.) With 100 estimators I have following scores(roc_score): train_data : 98.5 validation_data : 97.2

2.) With 500 estimators I have following scores(roc_score): train_data : 99.4 validation_data : 97.7

Acco. to above can we say model with 500 estimators works better. Or should I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators.


At first glance, your conclusion appears correct, but there are some important caveats to keep in mind.

First, what are the sizes of your training and validation sets? If your validation set is too small, then the observed difference may not be statistically significant.

Second, you should verify that your validation set is a representative sample. (i.e. it should come from the same distribution as the training set). If it's not representative, then it may give poor estimates of performance.

Third, when tuning hyperparameters, it's a good idea to split your dataset into three shards - training, validation, and testing. You can use the training and validation sets to find optimal hyperparameters (as you have done), then use the testing set to generate a performance estimate for the tuned model. If you trust the validation accuracy obtained during hyperparameter tuning, you are liable to a subtle form of overfitting where hyperparameters are specialized for the validation set.

Finally, if you have the computational resources, then it's always a good idea to evaluate accuracy with cross-validation rather than a train-test split. This will give you a more robust estimate of accuracy.

If you've checked all these boxes, then you have good reason to believe that 500 estimators is better than 100 estimators!

[S]hould I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators?

Yes, it's always a good idea to try many different configurations of hyperparameters. You can use scikit-learn's GridSearchCV or RandomizedSearchCV to easily run a search over the hyperparameter space.

  • $\begingroup$ Thanks for the answer. What if after tuning some hyperparameters there is increase in validation score but the test score decreases. How to deal with such situation. Sometimes this happens with me and It feels like all this while I have been tuning the model wrt validation set but suddenly its of no use. NOTE : Problem is of classification and data is highly imbalanced. $\endgroup$ – Sahil Oct 3 '19 at 6:38
  • $\begingroup$ When you experience an improvement in validation accuracy but a decrease in testing accuracy, that's a symptom of the subtle overfitting I mentioned. Basically, you've found hyperparameters which are specialized to your validation set rather than hyperparameters that perform best in general. Does that make sense? $\endgroup$ – zachdj Oct 3 '19 at 19:17
  • $\begingroup$ Regarding your note, gradient boosting is a good technique for dealing with imbalanced datasets :) Just make sure your training, validation, and test samples are stratified $\endgroup$ – zachdj Oct 3 '19 at 19:20

Just to add some general thoughts to the other answers. Gradient boosting is fairly robust to overfitting through increasing the number of trees. Increasing the number of trees is expected to increase the performance if the learning rate is small. It is therefore generally considered best to set the number of trees through early stopping instead of treating them like other hyperparameters.

You would set a small learning rate (something $\eta <0.1$) and the number of trees to a large value, and stop adding trees once you don't see any more improvements on a separate validation set.

As a last optional step, since adding more and more trees results in smaller and smaller gains in performance, once you have found a satisfactory model and want to put it in production, you can analyze how much you can reduce the number of trees without significantly reducing the performance. You can then trim the number of trees to speed up computation in production if speed is an issue. This idea is often also used for Random Forests because here adding more trees should never decrease performance.

  • $\begingroup$ Nice Insights, Thanks. $\endgroup$ – Sahil Oct 3 '19 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.