# Tag Info

Accepted

### Lightgbm vs xgboost vs catboost

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do ...
• 1,874
Accepted

### L1 & L2 Regularization in Light GBM

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...
• 10.8k

### Differences between class_weight and scale_pos weight in LightGBM

You can achieve the same results by using either class_weight, scale_pos_weight and ...

### LightGBM gives different results (metrics) depending on the columns order

A possible explanation is this: When the order of the columns differ, there is a little difference in the procedure. What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns ...
Accepted

### Random Forest VS LightGBM

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a ...
• 1,093

### What is the proper way to use early stopping with cross-validation?

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...
• 10.8k

### How do GBM algorithms handle missing data?

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. https://github.com/microsoft/LightGBM/issues/2921 There are some options you can set ...
• 5,519
Accepted

### How to make LightGBM to suppress output?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, ...
• 1,527

### How to make LightGBM to suppress output?

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in ...
• 7,207

### What is the proper way to use early stopping with cross-validation?

I think some answers to (/comments about) related questions are well addressed in these posts: https://stats.stackexchange.com/q/402403 https://stats.stackexchange.com/q/361494 In my mind, the tldr ...
Accepted

### SHAP value analysis gives different feature importance on train and test set

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance) You are just comparing two different instances and getting ...
• 5,776

### L1 & L2 Regularization in Light GBM

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5 Gabriel Tseng, Author of ...

### How to make LightGBM to suppress output?

Solution for sklearn API (checked on v3.3.0): ...
• 151

### handling missing values for LightGBM model

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...
• 10.8k

### How to make LightGBM to suppress output?

Follow these points. Use verbose= False in fit method. Use verbose= -100 when you call the ...
Accepted

### Light GBM Regressor, L1 & L2 Regularization and Feature Importances

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without ...
• 7,207

### Lightgbm confidence interval

If you are looking for a statistical trick, I don't know, but Recently Andrew NG team recently published about NGBoost. NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a ...
• 5,776

### Correct interpretation of summary_plot shap graph

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is $$E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S]$$ averaged ...
• 321
Accepted

### What is Pruning & Truncation in Decision Trees?

Your understanding is correct. xgboost has nice explanation in the docs. Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
• 814

### Light GBM Regressor, L1 & L2 Regularization and Feature Importances

Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here: These lightGBM L1 and L2 regularization ...

### LightGBM gives different results (metrics) depending on the columns order

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split ...
• 73
Accepted

### How is the "base value" of SHAP values calculated?

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With ...
• 10.8k

### LightGBM gives different results (metrics) depending on the columns order

Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for ...
Accepted

### How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example: <...
• 1,714
Accepted

### What does the repeated message "No further splits with positive gain, best gain: -inf" mean?

I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop. From Github's LightGBM issues we can find that: What's the meaning of "No ...

### Extremely high gain with LightGBM

Its definitely not regularization since default values are 0 (check here) n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND ...
• 5,519

### Small number of estimators in gradient boosting

First what is n_estimators: n_estimatorsinteger, optional (default=10) The number of trees in the forest. Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit ...
• 5,776
Accepted

### How to use r2-score as a loss function in LightGBM?

$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual. (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize ...
• 10.8k