I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?

  • 2
    $\begingroup$ I get the impression that it is more or less all LGBM now on Kaggle. XGB will be tested on odd chance that it performs better. CB is not used. I don't think there is much intuition to be gained more than that people use what gives the best results. $\endgroup$ – Simon Larsson Apr 19 '19 at 6:56
  • $\begingroup$ Recently there was a post about this towardsdatascience.com/… $\endgroup$ – Carlos Mougan Aug 20 '20 at 7:57
  • $\begingroup$ recently hahaha thanks! $\endgroup$ – David Masip Aug 20 '20 at 8:44

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but rather through histogram approximations. The result is a slight decrease in predictive performance for a much larger speed increase in training. This means more opportunity for feature engineering/experimentation/model tuning (all of which are key to winning Kaggle competitions) which inevitably yields larger increases in predictive performance (despite using histogram approximations).

CatBoost is not used as much because on average, it it found to be much slower than LightGBM. That being said, CatBoost is different in its implementation of gradient boosting which at times can give slightly more accurate predictions, in particular if you have large amounts of categorical features. I have never used CatBoost and so I encourage you to read that paper. Regardless, because rapid experimentation is vital in Kaggle competitions, LightGBM tends to be the go to algorithm when first creating strong base learners.

In general, it is important to note that a large amount of approaches I've seen involve combining all three boosting algorithms in a model stack (i.e. ensembling). LightGBM, CatBoost, and XGBoost might be thrown together as three base learners and then combined via. a GLM or neural network. This is done to really squeeze out decimal places on the leaderboard and so I doubt there is any theoretical (or practical) justification for it besides competitions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.