Skip to main content
33 votes
Accepted

Lightgbm vs xgboost vs catboost

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
aranglol's user avatar
  • 2,206
29 votes
Accepted

L1 & L2 Regularization in Light GBM

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
Ben Reiniger's user avatar
  • 12k
14 votes

Differences between class_weight and scale_pos weight in LightGBM

You can achieve the same results by using either class_weight, scale_pos_weight and ...
Mayank Mahawar's user avatar
11 votes
Accepted

Random Forest VS LightGBM

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a ...
Yohanes Alfredo's user avatar
11 votes

LightGBM gives different results (metrics) depending on the columns order

A possible explanation is this: When the order of the columns differ, there is a little difference in the procedure. What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns ...
Juan Esteban de la Calle's user avatar
9 votes
Accepted

How to make LightGBM to suppress output?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, ...
bradS's user avatar
  • 1,615
8 votes

How to make LightGBM to suppress output?

Solution for sklearn API (checked on v3.3.0): ...
banderlog013's user avatar
8 votes

What is the proper way to use early stopping with cross-validation?

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...
Ben Reiniger's user avatar
  • 12k
7 votes

How do GBM algorithms handle missing data?

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. https://github.com/microsoft/LightGBM/issues/2921 There are some options you can set ...
Noah Weber's user avatar
  • 5,699
6 votes

How to make LightGBM to suppress output?

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in ...
Peter's user avatar
  • 7,526
6 votes

What is the proper way to use early stopping with cross-validation?

I think some answers to (/comments about) related questions are well addressed in these posts: https://stats.stackexchange.com/q/402403 https://stats.stackexchange.com/q/361494 In my mind, the tldr ...
It'sRecreational's user avatar
5 votes
Accepted

SHAP value analysis gives different feature importance on train and test set

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance) You are just comparing two different instances and getting ...
Carlos Mougan's user avatar
5 votes
Accepted

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without ...
Peter's user avatar
  • 7,526
5 votes

L1 & L2 Regularization in Light GBM

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5 Gabriel Tseng, Author of ...
Joshua1990's user avatar
5 votes

Lightgbm confidence interval

If you are looking for a statistical trick, I don't know, but Recently Andrew NG team recently published about NGBoost. NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a ...
Carlos Mougan's user avatar
5 votes

handling missing values for LightGBM model

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...
Ben Reiniger's user avatar
  • 12k
4 votes

Light GBM Regressor, L1 & L2 Regularization and Feature Importances

Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here: These lightGBM L1 and L2 regularization ...
Kevin2342's user avatar
4 votes

How to make LightGBM to suppress output?

Follow these points. Use verbose= False in fit method. Use verbose= -100 when you call the ...
bharat bhimshetty's user avatar
4 votes

What is the proper way to use early stopping with cross-validation?

I once wondered the same in case of LightGBM and got this answer from its creator, Guolin Ke: I think in both XGBoost and LightGBM, the CV will use the average scores from all folds, and use this for ...
mirekphd's user avatar
  • 143
4 votes

Optuna Median Pruner n_warmup_steps

The steps in n_warmup_steps refer to the incremental steps taken during gradient decent. So with ...
OliverHennhoefer's user avatar
3 votes

Correct interpretation of summary_plot shap graph

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is $$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$ averaged ...
Andrey Popov's user avatar
3 votes
Accepted

What is Pruning & Truncation in Decision Trees?

Your understanding is correct. xgboost has nice explanation in the docs. Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
Piotr Rarus's user avatar
3 votes

LightGBM gives different results (metrics) depending on the columns order

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split ...
gbdata's user avatar
  • 73
3 votes
Accepted

How is the "base value" of SHAP values calculated?

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With ...
Ben Reiniger's user avatar
  • 12k
3 votes

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

Yes, you can train XGBoost in parallel using the Dask backend. Short Solution Training XGBoost in parallel with Dask requires 2 changes in your code: substitute ...
rrpelgrim's user avatar
3 votes
Accepted

Online Learning/Continual Learning for tree-based Algorithms

This is a really good question for which I will give you a theoretical result; in particular, I am not aware of any specific implementation in any programming language. The concept of incremental ...
Eduard's user avatar
  • 669
2 votes

LightGBM gives different results (metrics) depending on the columns order

Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for ...
Beverly Wang's user avatar
2 votes
Accepted

How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example: <...
Dan Scally's user avatar
  • 1,764
2 votes

Math Behind GOSS (Gradient-Based One Side Sampling)?

Wang et al., (2019) have provided a nice and clear explanation. Please, check out their paper to find the answer you are looking for: Part II. BAYESIAN OPTIMIZED LIGHTGBM Section: A. The Principle of ...
Bruno Ambrozio's user avatar
2 votes

Read back a saved LGBMClassifier model

Use clf_fs.predict instead. Reference link
Victor Dalla's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible