31 votes
Accepted

Lightgbm vs xgboost vs catboost

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
  • 1,864
20 votes
Accepted

L1 & L2 Regularization in Light GBM

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
  • 10.6k
11 votes

Differences between class_weight and scale_pos weight in LightGBM

You can achieve the same results by using either class_weight, scale_pos_weight and ...
9 votes

LightGBM gives different results (metrics) depending on the columns order

A possible explanation is this: When the order of the columns differ, there is a little difference in the procedure. What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns ...
8 votes

What is the proper way to use early stopping with cross-validation?

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...
  • 10.6k
7 votes
Accepted

How to make LightGBM to suppress output?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, ...
  • 1,527
7 votes
Accepted

Random Forest VS LightGBM

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a ...
6 votes

How do GBM algorithms handle missing data?

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. https://github.com/microsoft/LightGBM/issues/2921 There are some options you can set ...
  • 5,459
6 votes

How to make LightGBM to suppress output?

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in ...
  • 7,124
6 votes

What is the proper way to use early stopping with cross-validation?

I think some answers to (/comments about) related questions are well addressed in these posts: https://stats.stackexchange.com/q/402403 https://stats.stackexchange.com/q/361494 In my mind, the tldr ...
5 votes

L1 & L2 Regularization in Light GBM

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5 Gabriel Tseng, Author of ...
5 votes

handling missing values for LightGBM model

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...
  • 10.6k
4 votes
Accepted

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without ...
  • 7,124
4 votes

How to make LightGBM to suppress output?

Solution for sklearn API (checked on v3.3.0): ...
4 votes
Accepted

SHAP value analysis gives different feature importance on train and test set

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance) You are just comparing two different instances and getting ...
4 votes

Lightgbm confidence interval

If you are looking for a statistical trick, I don't know, but Recently Andrew NG team recently published about NGBoost. NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a ...
3 votes
Accepted

What is Pruning & Truncation in Decision Trees?

Your understanding is correct. xgboost has nice explanation in the docs. Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
3 votes

Correct interpretation of summary_plot shap graph

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is $$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$ averaged ...
3 votes

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here: These lightGBM L1 and L2 regularization ...
3 votes

How to make LightGBM to suppress output?

Follow these points. Use verbose= False in fit method. Use verbose= -100 when you call the ...
3 votes

LightGBM gives different results (metrics) depending on the columns order

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split ...
  • 73
3 votes
Accepted

How is the "base value" of SHAP values calculated?

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With ...
  • 10.6k
2 votes

LightGBM gives different results (metrics) depending on the columns order

Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for ...
2 votes
Accepted

How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example: <...
  • 1,714
2 votes

Read back a saved LGBMClassifier model

Use clf_fs.predict instead. Reference link
2 votes

Extremely high gain with LightGBM

Its definitely not regularization since default values are 0 (check here) n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND ...
  • 5,459
2 votes
Accepted

What does the repeated message "No further splits with positive gain, best gain: -inf" mean?

I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop. From Github's LightGBM issues we can find that: What's the meaning of "No ...
2 votes

Small number of estimators in gradient boosting

First what is n_estimators: n_estimatorsinteger, optional (default=10) The number of trees in the forest. Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit ...
2 votes
Accepted

How to use r2-score as a loss function in LightGBM?

$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual. (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize ...
  • 10.6k
2 votes

What is the proper way to use early stopping with cross-validation?

I once wondered the same in case of LightGBM and got this answer from its creator, Guolin Ke: I think in both XGBoost and LightGBM, the CV will use the average scores from all folds, and use this for ...
  • 123

Only top scored, non community-wiki answers of a minimum length are eligible