31
votes
Accepted
Lightgbm vs xgboost vs catboost
On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
- 1,874
20
votes
Accepted
L1 & L2 Regularization in Light GBM
First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
- 10.8k
11
votes
Differences between class_weight and scale_pos weight in LightGBM
You can achieve the same results by using either class_weight, scale_pos_weight and ...
- 111
9
votes
LightGBM gives different results (metrics) depending on the columns order
A possible explanation is this:
When the order of the columns differ, there is a little difference in the procedure.
What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns ...
- 2,222
8
votes
Accepted
Random Forest VS LightGBM
RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a ...
- 1,093
8
votes
What is the proper way to use early stopping with cross-validation?
I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...
- 10.8k
7
votes
How do GBM algorithms handle missing data?
LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. https://github.com/microsoft/LightGBM/issues/2921
There are some options you can set ...
- 5,519
7
votes
Accepted
How to make LightGBM to suppress output?
As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here).
However, ...
- 1,527
6
votes
How to make LightGBM to suppress output?
To suppress (most) output from LightGBM, the following parameter can be set.
Suppress warnings: 'verbose': -1 must be specified in ...
- 7,207
6
votes
What is the proper way to use early stopping with cross-validation?
I think some answers to (/comments about) related questions are well addressed in these posts:
https://stats.stackexchange.com/q/402403
https://stats.stackexchange.com/q/361494
In my mind, the tldr ...
- 146
5
votes
Accepted
SHAP value analysis gives different feature importance on train and test set
Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance)
You are just comparing two different instances and getting ...
- 5,776
5
votes
L1 & L2 Regularization in Light GBM
In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5
Gabriel Tseng, Author of ...
- 71
5
votes
5
votes
handling missing values for LightGBM model
The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...
- 10.8k
4
votes
How to make LightGBM to suppress output?
Follow these points.
Use verbose= False in fit method.
Use verbose= -100 when you call the ...
4
votes
Accepted
Light GBM Regressor, L1 & L2 Regularization and Feature Importances
With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without ...
- 7,207
4
votes
Lightgbm confidence interval
If you are looking for a statistical trick, I don't know, but
Recently Andrew NG team recently published about NGBoost.
NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a ...
- 5,776
3
votes
Correct interpretation of summary_plot shap graph
I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is
$$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$
averaged ...
- 321
3
votes
Accepted
What is Pruning & Truncation in Decision Trees?
Your understanding is correct. xgboost has nice explanation in the docs.
Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
- 814
3
votes
Light GBM Regressor, L1 & L2 Regularization and Feature Importances
Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here:
These lightGBM L1 and L2 regularization ...
- 31
3
votes
LightGBM gives different results (metrics) depending on the columns order
While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split ...
- 73
3
votes
Accepted
How is the "base value" of SHAP values calculated?
As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With ...
- 10.8k
2
votes
LightGBM gives different results (metrics) depending on the columns order
Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for ...
- 21
2
votes
Accepted
How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually
Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:
<...
- 1,714
2
votes
Accepted
What does the repeated message "No further splits with positive gain, best gain: -inf" mean?
I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop.
From Github's LightGBM issues we can find that:
What's the meaning of "No ...
2
votes
2
votes
Extremely high gain with LightGBM
Its definitely not regularization since default values are 0 (check here)
n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND ...
- 5,519
2
votes
Small number of estimators in gradient boosting
First what is n_estimators:
n_estimatorsinteger, optional (default=10)
The number of trees in the forest.
Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit ...
- 5,776
2
votes
Accepted
How to use r2-score as a loss function in LightGBM?
$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual. (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize ...
- 10.8k
2
votes
What is the proper way to use early stopping with cross-validation?
I once wondered the same in case of LightGBM and got this answer from its creator, Guolin Ke:
I think in both XGBoost and LightGBM, the CV will use the average
scores from all folds, and use this for ...
- 123
Only top scored, non community-wiki answers of a minimum length are eligible