30

On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but ...


17

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So using both isn't unprecedented. Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose. Tree complexity can be ...


8

A possible explanation is this: When the order of the columns differ, there is a little difference in the procedure. What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training. The selections of these columns is done randomly: Let's say your dataset has 20 columns. The ...


7

You can achieve the same results by using either class_weight, scale_pos_weight and is_unbalanced for binary classification on unbalanced dataset. Setting class_weight = {0: (number of negative samples / number of positive samples), 1: (number of positive samples / number of negative samples)} is the same as setting is_unbalance = True or ...


7

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model's ideal number of estimators compares to those of the cv iterations. For example, if your validation performance rises sharply with additional estimators, ...


6

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a properly-tuned GBM. Another advantage is that you do not need to care a lot about parameter. You can compare the number of parameter for randomforest model and ...


6

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, LightGBM may still return other warnings - e.g. No further splits with positive gain. This can be suppressed as follows (source: here ): lgb_train = lgb.Dataset(X_train, y_train, params={'verbose': -1}, free_raw_data=False) lgb_eval = lgb.Dataset(...


5

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in params={}. Suppress output of training iterations: verbose_eval=False must be specified in the train{} parameter. Minimal example: params = { 'objective': 'regression', 'learning_rate' : 0.9, ...


5

In this medium post, you can find a concise and very clear explanation regarding these parameters https://medium.com/@gabrieltseng/gradient-boosting-and-xgboost-c306c1bcfaf5 Gabriel Tseng, Author of the blogpost: "These two regularization terms have different effects on the weights; L2 regularization (controlled by the lambda term) encourages the weights to ...


5

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take the left branch, and so your model has (slightly) less capacity. That may be a good or a bad thing, depending on where you land on the bias-variance curve. So, ...


4

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Section 3.2 of this reference explains it. There are some options you can set such as usemissing=false, which disables handling for missing values. You can also use the zeroas_missing option to change behavior. GitHub


3

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is ...


3

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without regularization. You don't need to exclude any features since the purpose of shrinking is to use features according to their importance (this happens automatically). In your case the top two features seem ...


3

I think some answers to (/comments about) related questions are well addressed in these posts: https://stats.stackexchange.com/q/402403 https://stats.stackexchange.com/q/361494 In my mind, the tldr summary as it relates to your question is that after cross validation one could (or maybe should) retrain a model using a single very large training set, with ...


2

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance) You are just comparing two different instances and getting different results. This is normal and can happen in train and test set. This doesn't mean also that your train and test set have bad split, they could be good ...


2

Your understanding is correct. xgboost has nice explanation in the docs. Reading the original papers is always great idea. Here's one for LGBM and here's one for XGBoost. As for dessert, here's catboost paper. Boosting is by far one of the most important concepts in hard machine learning. It's good to know it by heart.


2

Here's a link to a good answer for the follow up question of "should you use both L1 and L2 regularization terms?" Summarized briefly here: These lightGBM L1 and L2 regularization parameters are related leaf scores, not feature weights. The regularization terms will reduce the complexity of a model (similar to most regularization efforts) but ...


2

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is $$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$ averaged over all possible subsets of features $S$, $i \notin S$. Here $f(x)$ is the prediction of the model for inputs $x$. Figure 2 from the preprint is a good ...


2

Its definitely not regularization since default values are 0 (check here) n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND columns (again depending on the parameters) so it could be that you took some unfortunate combination which should level out with higher number of n_estimators. ...


2

I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop. From Github's LightGBM issues we can find that: What's the meaning of "No further splits with positive gain, best gain: -inf" message? It means the learning of tree in current iteration should be stop, due to cannot split any more....


2

First what is n_estimators: n_estimatorsinteger, optional (default=10) The number of trees in the forest. Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit several trees and then they average(ensemble) them. If you have n_estimators=1, means that you just have one tree, if you have n_estimators=3 means that you ...


2

$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual. (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize outliers less.)


2

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification). With force_plot, you actually pass your desired base value as the first parameter; in that notebook's case it is explainer.expected_value[1], the average of the second class. https://github....


2

If you are looking for a statistical trick, I don't know, but Recently Andrew NG team recently published about NGBoost. NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a modular boosting algorithm for probabilistic predictions. In this Towards Data Science toy example you can see how to use the Python API: Quoting the TDS author: ...


2

Gradient boosting can be applied to any base model, so doing it with a Quinlan-family decision tree (which allow for such higher-arity splits for categorical features) should make this possible. However, all implementations of gradient boosted trees that I know of (and certainly XGBoost, CatBoost, LightGBM) all use CART as their tree model, so you won't get ...


2

There is no free lunch among Catboost, XGBoost and LightGBM. In my experience, some cases I found that XGBoost outperform other, some cases for LightGBM, and the rest for CatBoost. So there is no exact the best model until you test them all in your dataset with doing hyper parameters tune to your model. The only think clearly better from both CatBoost and ...


2

Yes, you can train XGBoost in parallel using the Dask backend. Short Solution Training XGBoost in parallel with Dask requires 2 changes in your code: substitute dtrain = xgb.DMatrix(X_train, y_train) with dtrain = xgb.dask.DaskDMatrix(X_train, y_train) substitute xgb.train(params, dtrain, ...) with xgb.dask.train(client, params, dtrain, ...) Have a look ...


2

You are unlikely to get a useful answer without a lot more details as there are lots of things that could cause this. How many features and how many observations do you have? It is possible that you have massively overfit your training set: Did you do a lot of hyper parameter tuning on your model? when you fit the light GBM, did you cross validate your ...


1

Yes, there are decision tree algorithms using this criterion, e.g. see C4.5 algorithm, and it is also used in random forest classifiers. See, for example, the random forest classifier scikit learn documentation: criterion: string, optional (default=”gini”) The function to measure the quality of a split. Supported criteria are “gini” for the Gini ...


1

There are different ways to include categorical features, and in many of them a single leaf can combine multiple categories: With the label, target, or frequency encoding the categorical feature is effectively replaced by a numeric one, so a leaf can include multiple original categories naturally. Conversely, any numeric feature can be thought of as an ...


Only top voted, non community-wiki answers of a minimum length are eligible