67
votes
Accepted
GBM vs XGBOOST? Key differences?
Quote from the author of xgboost:
Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
- 4,296
61
votes
Accepted
Does XGBoost handle multicollinearity by itself?
Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
- 2,507
28
votes
Accepted
Why is xgboost so much faster than sklearn GradientBoostingClassifier?
Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In ...
- 1,366
28
votes
Does XGBoost handle multicollinearity by itself?
I was curious about this and made a few tests.
I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
- 381
27
votes
Does XGBoost handle multicollinearity by itself?
There is an answer from Tianqi Chen (2018).
This difference has an impact on a corner case in feature importance analysis: the correlated features.
Imagine two features perfectly correlated, ...
- 400
20
votes
GBM vs XGBOOST? Key differences?
In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
- 2,507
15
votes
GBM vs XGBOOST? Key differences?
One very important difference is xgboost has implemented DART, the dropout regularization for regression trees.
References
Rashmi, K. V., & Gilad-Bachrach, ...
- 1,350
12
votes
Accepted
How fit pairwise ranking models in XGBoost?
According to the XGBoost documentation, XGboost expects:
the examples of a same group to be consecutive examples,
a list with the size of each group (which you can set with ...
- 256
10
votes
Need help understanding xgboost's approximate split points proposal
I won't go into details but the following should help you grasp the idea.
They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ ...
- 1,366
8
votes
Need help understanding xgboost's approximate split points proposal
Just adding the algebraic part to @Winks answer:
The second equation should have its sign reversed, as in:
$$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - (-g_i/h_i)]^2 + constant =
\sum_{i=1}^n\frac{1}{2}...
- 1,357
7
votes
Can xgboost (or any other algorithm) give bad results with some bad features?
Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also ...
- 9,348
7
votes
What is init_score in lightGBM?
Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each ...
- 3,920
7
votes
Adding feature leads to worse results
To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect.
Adding new features is not always beneficial, because you increase the ...
- 1,490
6
votes
Accepted
How to determine if my GBM model is overfitting?
The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data.
Just by looking at the model ...
- 371
6
votes
How fit pairwise ranking models in XGBoost?
set_group is very important to ranking, because only the scores in one group are comparable.
You can sort data according to their scores in their own group.
For ...
- 61
6
votes
difference between logistic regression and binary logistic regression
binary:logistic is used for binary classification where the target variable takes binary output [0, 1]
reg:logistic is used for ...
- 509
6
votes
Decision Trees Nodes vs Leaves Definition
Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In ...
- 3,920
5
votes
Accepted
Does the performance of GBM methods profit from feature scaling?
Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree.
This post that elaborates on the topic, but mainly the issue is that ...
- 6,031
5
votes
Accepted
Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?
Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
- 6,082
4
votes
Fit Decision Tree to Gradient Boosted Trees for Interpretability
Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual ...
- 961
4
votes
Does XGBoost handle multicollinearity by itself?
A remark on Sandeep's answer:
Assuming 2 of your features are highly colinear (say equal 99% of time)
Indeed only 1 feature is selected at each split, but for the next split, the xgb can select the ...
- 41
4
votes
Accepted
Representing cyclical features as sin/cos components
as both components need to be equally weighted in order for the feature to make sense
That is not the case.
For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it ...
- 28.6k
4
votes
LightGBM - Why Exclusive Feature Bundling (EFB)?
I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of ...
- 490
3
votes
How to determine if my GBM model is overfitting?
I suggest training/testing your classifier on separate splits of the original dataset, and then printing a confusion matrix:
https://topepo.github.io/caret/measuring-performance.html#
This is a way ...
- 1,732
3
votes
Residuals in a gradient boosted classification
It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to ...
- 9,348
3
votes
Adding feature leads to worse results
Welcome to the site!
If I understand your question correctly you want to know why a model would perform worse when a new feature is added?
So every time you do feature engineering (add new columns, ...
- 2,382
3
votes
Accepted
How does the number of trees effect the prediction time in gradient boost classification trees?
The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take ...
- 3,920
3
votes
Accepted
LightGBM - Why Exclusive Feature Bundling (EFB)?
See this article for a bit more detail on how to better explain EFB. Here is a brief visual explanation from there with my own edits. I hope you can appreciate the high production quality of my ...
- 166
3
votes
Train classifier on balanced dataset and apply on imbalanced dataset?
Is this a problem?
No. not at all.
Will the model reproduce the trained target distribution from DS1 when
applied on DS2?
No, not necessarily. If
Balanced set DS1 is a good representative of ...
- 9,227
2
votes
Can xgboost (or any other algorithm) give bad results with some bad features?
I am hesitant to think of features as bad features. However, there are features more or less useful relative to other features.
In general pruning features are seen as a good best practice. ...
Only top scored, non community-wiki answers of a minimum length are eligible