Skip to main content
4 votes
Accepted

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

"Unbalanced" data are not a problem, unless you use unsuitable error measures... like accuracy, or precision, recall and the F1 (or any other Fbeta) score, all of which suffer from exactly ...
Stephan Kolassa's user avatar
3 votes

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

Your model might be alright, but certainly not with a default classification threshold. As you can see, you only detected 2 out of 123 events this way. ROC_AUC tends to be overoptimistic for this ...
dx2-66's user avatar
  • 696
3 votes

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

Your minority class is highly under-represented. I recommend not to proceed forward. My suggestion would be the following: 1.) Undersample the majority class 2.) Use SMOTE to oversample the minority ...
mewbie's user avatar
  • 109
3 votes
Accepted

What is Pruning & Truncation in Decision Trees?

Your understanding is correct. xgboost has nice explanation in the docs. Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
Piotr Rarus's user avatar
3 votes
Accepted

Reasons for a model predicting probability of being class 1 at x value

Not necessarily, while it can be the case that two observations belong to the same 'group' and end up in the same leaf node (and thus get the same predicted value) there can also be multiple groups of ...
Oxbowerce's user avatar
  • 7,657
3 votes
Accepted

Catboost not able to handle a very simple dataset?

"Traditional" tree models cannot extrapolate well outside the training data's range, so "I want to use shuffle = False for reasons beyond the scope ...
Ben Reiniger's user avatar
  • 12k
2 votes

What if root of a such tree is pruned in xgboost?

You'll be left with one-node trees. The loss reduction of a split is penalized by $\gamma$, but the root itself does not get pruned. This is fairly easy to test: ...
Ben Reiniger's user avatar
  • 12k
2 votes
Accepted

XGBoost - Imputing Vs keeping NaN

it is boring and repetitive to say - so sorry - but it depends what are you trying to predict and if it is make sense to do imputation at all. If you are trying to predict the car price probably it is ...
user702846's user avatar
2 votes
Accepted

Tree complexity and gamma parameter in xgboost

In the paragraph following equation (1): $T$ is the number of leaves in the tree. $\gamma$ is a hyperparameter that affects how much regularization occurs on the size (number of leaves) of the tree. ...
Ben Reiniger's user avatar
  • 12k
2 votes

What is "Missing" in output of plot_tree API of XGBoost

The "missing" at the different nodes are the observations for which the feature on which the split is made is missing. E.g. if the value for ...
Oxbowerce's user avatar
  • 7,657
2 votes

XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values

It may not be related to hyperparams per say. I think it has more to do with the nature of how xgboost is trained. XGBoost for regression tries to reduce the variance at every node. May when you have ...
Ashwiniku918's user avatar
  • 2,024
2 votes
Accepted

Why is monotonic constraint disabled when using MAE as an objective to LGBM?

MAE is an odd loss function for GBMs: the gradient is constant ($\pm1$), and the hessian all zeros, so the usual tree-training target of $-G/H$ (possibly with additional terms for regularization) ...
Ben Reiniger's user avatar
  • 12k
2 votes
Accepted

How is the weight of each new weak learner is calculated in Xgboost?

Your description sounds a bit more like AdaBoost than Gradient Boosting (XGBoost and others). AdaBoost trains its weak estimators on the original targets, just weighting/sampling the observations ...
Ben Reiniger's user avatar
  • 12k
1 vote

XGB predict_proba estimates don't match sum of leaves

When you call predict_proba in XGBoost, it returns the probability estimates calculated by averaging the predictions of all the ...
Multivac's user avatar
  • 3,009
1 vote

randomness in lightgbm model training

What can be the other factors adding randomness to the training process that I did not notice? By "adding randomness to the training process", I'm assuming that you mean things that could ...
James Lamb's user avatar
1 vote

How to reduce the false positives to improve the models performance?

Since you did not mention based on which metric your decision will be drawn (e.g. if model A has better F1 but model B is better on ROC-AUC, which would you pick?) nor much detail on your dataset, I ...
lpounng's user avatar
  • 1,092
1 vote

DART algorithm implementation. Converting mathematical notation to pseudocode

Well, I am not familiar with the DART algorithm, but I thought it might be a good task for ChatGPT, so I took the Latex source of the article you linked, and asked it to generate C# code for it: ...
noe's user avatar
  • 27k
1 vote

Why is gradient boosting better than random forest for unbalanced data?

Boosting is the method of creating ensemble by increasing the importance of wrongly predicted instances in each iteration. RandomForest works by creating ensemble using Bootstrap Aggregating which ...
Ashish Jain's user avatar
1 vote

Impact of many zeros in LightGBM Regressor training set

All five of the features are NA for each row. I impute the missing values with zero before I feed into the model. By setting them 0, the model will actually learn that the values are zero. I would ...
mehmat's user avatar
  • 33
1 vote
Accepted

Gradient tree boosting additive training

do they mean that $\sum\limits_{k=1}^{t-1} \omega(f_k) = constant$? Yes! From the paragraph preceding that: we use an additive strategy: fix what we have learned, and add one new tree at a time. ...
Ben Reiniger's user avatar
  • 12k
1 vote
Accepted

How does Catboost regressor deal with categorical features at predict time?

In a simplified way of putting it, we substitute the category id with the mean value of the training set target for this category. CatBoost implements some tricks like only using the preceding values ...
dx2-66's user avatar
  • 696
1 vote

GridSearch multiplying the number of trees in XGboost?

You have three classes; xgboost is building three one-vs-rest models, hence three times the trees. https://github.com/dmlc/xgboost/issues/806
Ben Reiniger's user avatar
  • 12k
1 vote
Accepted

How is the probability prediction of a binary classifier predicted

y is beta0 + beta1*x1 + beta2*x2 .....+ epsilong probability = ((e power y)/(1 + e power y)) ...
karteek menda's user avatar
1 vote

Example for Boosting

Please refer to this link in order to understand boosting with a worked out example. https://xgboost.readthedocs.io/en/latest/tutorials/model.html
Ashwiniku918's user avatar
  • 2,024
1 vote

Example for Boosting

I think what you actually ask is "how does boosting work". LightGBM or XGBoost are implementations of boosting algorithms. I like the article by Bühlmann and Hothorn. They provide a very ...
Peter's user avatar
  • 7,526
1 vote

Feature importance by random forest and boosting tree when two features are heavy correlated

Randomized sparse models: Random forest and boosting tree already have the feature (column) sampling, are two effects repeated? This question is unclear. Random Forest and GBDT are not randomized ...
Carlos Mougan's user avatar
1 vote
Accepted

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree? It's harder yes, but common. In the same tree it can not happen in the ...
Carlos Mougan's user avatar
1 vote

Multi-target regression tree with additional constraint

So there does not seem to be anything that is out of the box ready for this but, I found an example of someone doing something similar to what you want to do with a Random Forest. Here is the link: ...
Miguel Raevenswood's user avatar
1 vote

Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?

In case you hadn't seen the User Guide section for this method, the explanation there is pretty good: These fast estimators first bin the input samples X into ...
Ben Reiniger's user avatar
  • 12k
1 vote

Chossing between gradient boosting algorithms

I would say Catboost and lightgbm perform similarly and its purely a matter of choice. Some of my colleagues prefered ...
Yaroslaw Homenko's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible