3
votes
Accepted
Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?
"Unbalanced" data are not a problem, unless you use unsuitable error measures... like accuracy, or precision, recall and the F1 (or any other Fbeta) score, all of which suffer from exactly ...
- 996
3
votes
Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?
Your model might be alright, but certainly not with a default classification threshold. As you can see, you only detected 2 out of 123 events this way. ROC_AUC tends to be overoptimistic for this ...
- 656
3
votes
Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?
Your minority class is highly under-represented. I recommend not to proceed forward.
My suggestion would be the following:
1.) Undersample the majority class
2.) Use SMOTE to oversample the minority ...
- 109
3
votes
Accepted
What is Pruning & Truncation in Decision Trees?
Your understanding is correct. xgboost has nice explanation in the docs.
Reading the original papers is always great idea. Here's one for LGBM and here's one for ...
- 814
3
votes
Accepted
Reasons for a model predicting probability of being class 1 at x value
Not necessarily, while it can be the case that two observations belong to the same 'group' and end up in the same leaf node (and thus get the same predicted value) there can also be multiple groups of ...
- 6,872
3
votes
Accepted
Catboost not able to handle a very simple dataset?
"Traditional" tree models cannot extrapolate well outside the training data's range, so "I want to use shuffle = False for reasons beyond the scope ...
- 10.8k
2
votes
What if root of a such tree is pruned in xgboost?
You'll be left with one-node trees. The loss reduction of a split is penalized by $\gamma$, but the root itself does not get pruned. This is fairly easy to test:
...
- 10.8k
2
votes
Accepted
XGBoost - Imputing Vs keeping NaN
it is boring and repetitive to say - so sorry - but it depends what are you trying to predict and if it is make sense to do imputation at all. If you are trying to predict the car price probably it is ...
- 313
2
votes
XGBoost regression scale invariant? 0 depth trees for target variable with small (1E-7) values
It may not be related to hyperparams per say. I think it has more to do with the nature of how xgboost is trained. XGBoost for regression tries to reduce the variance at every node. May when you have ...
- 1,799
2
votes
Accepted
Why is monotonic constraint disabled when using MAE as an objective to LGBM?
MAE is an odd loss function for GBMs: the gradient is constant ($\pm1$), and the hessian all zeros, so the usual tree-training target of $-G/H$ (possibly with additional terms for regularization) ...
- 10.8k
1
vote
Why is gradient boosting better than random forest for unbalanced data?
Boosting is the method of creating ensemble by increasing the importance of wrongly predicted instances in each iteration.
RandomForest works by creating ensemble using Bootstrap Aggregating which ...
- 139
1
vote
Accepted
Gradient tree boosting additive training
do they mean that
$\sum\limits_{k=1}^{t-1} \omega(f_k) = constant$?
Yes! From the paragraph preceding that:
we use an additive strategy: fix what we have learned, and add one new tree at a time.
...
- 10.8k
1
vote
Accepted
How does Catboost regressor deal with categorical features at predict time?
In a simplified way of putting it, we substitute the category id with the mean value of the training set target for this category. CatBoost implements some tricks like only using the preceding values ...
- 656
1
vote
GridSearch multiplying the number of trees in XGboost?
You have three classes; xgboost is building three one-vs-rest models, hence three times the trees.
https://github.com/dmlc/xgboost/issues/806
- 10.8k
1
vote
Accepted
How is the probability prediction of a binary classifier predicted
y is beta0 + beta1*x1 + beta2*x2 .....+ epsilong
probability = ((e power y)/(1 + e power y))
...
1
vote
Feature importance by random forest and boosting tree when two features are heavy correlated
Randomized sparse models: Random forest and boosting tree already have the feature (column) sampling, are two effects repeated?
This question is unclear. Random Forest and GBDT are not randomized ...
- 5,776
1
vote
Accepted
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree
If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree?
It's harder yes, but common. In the same tree it can not happen in the ...
- 5,776
1
vote
Why is HistGradientBoostingRegressor in sklearn so fast and low on memory?
In case you hadn't seen the User Guide section for this method, the explanation there is pretty good:
These fast estimators first bin the input samples X into ...
- 10.8k
1
vote
Accepted
Tree complexity and gamma parameter in xgboost
In the paragraph following equation (1):
$T$ is the number of leaves in the tree.
$\gamma$ is a hyperparameter that affects how much regularization occurs on the size (number of leaves) of the tree.
...
- 10.8k
1
vote
Chossing between gradient boosting algorithms
I would say Catboost and lightgbm perform similarly and its purely a matter of choice.
Some of my colleagues prefered ...
- 538
1
vote
What is "Missing" in output of plot_tree API of XGBoost
The "missing" at the different nodes are the observations for which the feature on which the split is made is missing. E.g. if the value for ...
- 6,872
Only top scored, non community-wiki answers of a minimum length are eligible