I am working on the Kaggle home loan model and interestingly enough, the GradientBoostClassifier has a considerably better score than XGBClassifier. At the same time it seems to not overfit as much. (note, I am running both algos with default settings). From what I've been reading XGBClassifier is the same as GradientBoostClassifier, just much faster and more robust. Therefore I am now confused on why would XGB overfit so much more than GradientBoostClassifier, when it should do the contrary? What would be a good reason why this is happening?

  • 1
    $\begingroup$ Well, not really, I know what the differences are. I am more interested in what context GBM can have a higher score and overfit at a lesser degree than XGBoost. Knowing the difference, it's a bit strange for this to happen. Which is what I am observing. $\endgroup$
    – callmeGuy
    Jun 19 '20 at 12:13
  • $\begingroup$ It does not make sense to compare such algos without tuning. $\endgroup$
    – Michael M
    Jun 19 '20 at 16:50
  • $\begingroup$ I think it makes a lot of sense, you can see from the beginning what is worth spending time on hyper parameter tuning and what is not worth spending time on $\endgroup$
    – callmeGuy
    Jun 19 '20 at 16:54

GradientBoostClassifier is slower but more precise. In your case, it could be finding a better model without suffering from overfitting.

Here are some of the main differences you are looking for.

XGBClassifier was designed to be faster. However, XGBClassifier takes a few shortcuts in order to run faster. For example, to save time, XGBClassifier will use an approximation on the splits, not spend so much time on calculating and evaluating the best splits. Usually XGB results will be close and you won't see much difference, but your case could be an exception.

If you are still curious, you can try changing some parameters in the XGBClassifier model, such as the tree_method='exact' or you can modify the sketch_eps parameter in XGBClassifier to more closely match the GradientBoostClassifier results. Of course, this will slow down the XGBClassifier model as a result.

sketch_eps [default=0.03]

Only used for tree_method=approx.

This roughly translates into O(1 / sketch_eps) number of bins. Compared to directly select number of bins, this comes with theoretical guarantee with sketch accuracy.

Usually user does not have to tune this. But consider setting to a lower number for more accurate enumeration of split candidates.

range: (0, 1)

  • $\begingroup$ ya, this answer helps. Also, if I Grid Search, then XGBoost does do better than GB. However, as you said, it takes a bit longer, though in the same time it's easier to use a GPU on XGBoost. $\endgroup$
    – callmeGuy
    Jun 19 '20 at 12:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.