I've seen that in Kaggle competitions people are using lightgbms where they used to use xgboost. My question is: when would you rather use xgboost instead of lightgbm? What about catboost?
$\begingroup$
$\endgroup$
3
asked Apr 19, 2019 at 6:08
-
3$\begingroup$ I get the impression that it is more or less all LGBM now on Kaggle. XGB will be tested on odd chance that it performs better. CB is not used. I don't think there is much intuition to be gained more than that people use what gives the best results. $\endgroup$– Simon LarssonApr 19, 2019 at 6:56
-
$\begingroup$ Recently there was a post about this towardsdatascience.com/… $\endgroup$– Carlos MouganAug 20, 2020 at 7:57
-
$\begingroup$ recently hahaha thanks! $\endgroup$– David MasipAug 20, 2020 at 8:44
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but rather through histogram approximations. The result is a slight decrease in predictive performance for a much larger speed increase in training. This means more opportunity for feature engineering/experimentation/model tuning (all of which are key to winning Kaggle competitions) which inevitably yields larger increases in predictive performance (despite using histogram approximations).
CatBoost is not used as much because on average, it it found to be much slower than LightGBM. That being said, CatBoost is different in its implementation of gradient boosting which at times can give slightly more accurate predictions, in particular if you have large amounts of categorical features. I have never used CatBoost and so I encourage you to read that paper. Regardless, because rapid experimentation is vital in Kaggle competitions, LightGBM tends to be the go to algorithm when first creating strong base learners.
In general, it is important to note that a large amount of approaches I've seen involve combining all three boosting algorithms in a model stack (i.e. ensembling). LightGBM, CatBoost, and XGBoost might be thrown together as three base learners and then combined via. a GLM or neural network. This is done to really squeeze out decimal places on the leaderboard and so I doubt there is any theoretical (or practical) justification for it besides competitions.
