32
votes
Accepted
Lightgbm vs xgboost vs catboost
On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
- 1,954
27
votes
Accepted
L1 & L2 Regularization in Light GBM
First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
- 11.2k
13
votes
Differences between class_weight and scale_pos weight in LightGBM
You can achieve the same results by using either class_weight, scale_pos_weight and ...
- 131
11
votes
Accepted
Random Forest VS LightGBM
RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a ...
- 1,123
11
votes
LightGBM gives different results (metrics) depending on the columns order
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns ...
- 2,262
8
votes
8
votes
Accepted
How to make LightGBM to suppress output?
As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here).
However, ...
- 1,567
8
votes
What is the proper way to use early stopping with cross-validation?
I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...
- 11.2k
7
votes
How do GBM algorithms handle missing data?
LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. https://github.com/microsoft/LightGBM/issues/2921
There are some options you can set ...
- 5,629
6
votes
How to make LightGBM to suppress output?
To suppress (most) output from LightGBM, the following parameter can be set.
Suppress warnings: 'verbose': -1 must be specified in ...
- 7,366
6
votes
What is the proper way to use early stopping with cross-validation?
I think some answers to (/comments about) related questions are well addressed in these posts:
https://stats.stackexchange.com/q/402403
https://stats.stackexchange.com/q/361494
In my mind, the tldr ...
- 146
5
votes
Accepted
SHAP value analysis gives different feature importance on train and test set
Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance)
You are just comparing two different instances and getting ...
- 6,082
5
votes
Accepted
Light GBM Regressor, L1 & L2 Regularization and Feature Importances
With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without ...
- 7,366
5
votes
L1 & L2 Regularization in Light GBM
In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5
Gabriel Tseng, Author of ...
- 71
5
votes
Lightgbm confidence interval
If you are looking for a statistical trick, I don't know, but
Recently Andrew NG team recently published about NGBoost.
NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a ...
- 6,082
5
votes
handling missing values for LightGBM model
The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...
- 11.2k
4
votes
Light GBM Regressor, L1 & L2 Regularization and Feature Importances
Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here:
These lightGBM L1 and L2 regularization ...
- 41
4
votes
How to make LightGBM to suppress output?
Follow these points.
Use verbose= False in fit method.
Use verbose= -100 when you call the ...
4
votes
Optuna Median Pruner n_warmup_steps
The steps in n_warmup_steps refer to the incremental steps taken during gradient decent. So with ...
- 338
3
votes
Correct interpretation of summary_plot shap graph
I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is
$$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$
averaged ...
- 321
3
votes
Accepted
What is Pruning & Truncation in Decision Trees?
Your understanding is correct. xgboost has nice explanation in the docs.
Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
- 814
3
votes
LightGBM gives different results (metrics) depending on the columns order
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split ...
- 73
3
votes
Accepted
How is the "base value" of SHAP values calculated?
As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With ...
- 11.2k
3
votes
What is the proper way to use early stopping with cross-validation?
I once wondered the same in case of LightGBM and got this answer from its creator, Guolin Ke:
I think in both XGBoost and LightGBM, the CV will use the average
scores from all folds, and use this for ...
- 133
2
votes
LightGBM gives different results (metrics) depending on the columns order
Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for ...
- 21
2
votes
Accepted
How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually
Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:
<...
- 1,744
2
votes
Accepted
What does the repeated message "No further splits with positive gain, best gain: -inf" mean?
I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop.
From Github's LightGBM issues we can find that:
What's the meaning of "No ...
2
votes
Extremely high gain with LightGBM
Its definitely not regularization since default values are 0 (check here)
n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND ...
- 5,629
2
votes
Math Behind GOSS (Gradient-Based One Side Sampling)?
Wang et al., (2019) have provided a nice and clear explanation. Please, check out their paper to find the answer you are looking for:
Part II. BAYESIAN OPTIMIZED LIGHTGBM
Section: A. The Principle of ...
- 121
2
votes
Only top scored, non community-wiki answers of a minimum length are eligible