# Tag Info

Accepted

### GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
• 4,046
Accepted

### Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
• 2,347
Accepted

### Why is xgboost so much faster than sklearn GradientBoostingClassifier?

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In ...
• 1,316

### Does XGBoost handle multicollinearity by itself?

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
• 331

### Does XGBoost handle multicollinearity by itself?

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, ...
• 330

### GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
• 2,347

### GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
• 1,320
Accepted

### How fit pairwise ranking models in XGBoost?

According to the XGBoost documentation, XGboost expects: the examples of a same group to be consecutive examples, a list with the size of each group (which you can set with ...
• 246
Accepted

### Kappa near to 60% in unbalanced (1:10) data set

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing ...
• 621

### Need help understanding xgboost's approximate split points proposal

I won't go into details but the following should help you grasp the idea. They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ ...
• 1,316

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the ...
• 1,470

### What is init_score in lightGBM?

Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each ...
• 3,780
Accepted

### How to determine if my GBM model is overfitting?

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model ...
• 371

### Need help understanding xgboost's approximate split points proposal

Just adding the algebraic part to @Winks answer: The second equation should have its sign reversed, as in: \sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - (-g_i/h_i)]^2 + constant = \sum_{i=1}^n\frac{1}{2}...
• 1,277

### difference between logistic regression and binary logistic regression

binary:logistic is used for binary classification where the target variable takes binary output [0, 1] reg:logistic is used for ...
• 509

### Can xgboost (or any other algorithm) give bad results with some bad features?

Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also ...
• 9,028

### Decision Trees Nodes vs Leaves Definition

Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In ...
• 3,780

### How fit pairwise ranking models in XGBoost?

set_group is very important to ranking, because only the scores in one group are comparable. You can sort data according to their scores in their own group. For ...
• 51
Accepted

### Does the performance of GBM methods profit from feature scaling?

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that ...
• 5,664
Accepted

### Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
• 5,591

### Fit Decision Tree to Gradient Boosted Trees for Interpretability

Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual ...
• 961
Accepted

### Representing cyclical features as sin/cos components

as both components need to be equally weighted in order for the feature to make sense That is not the case. For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it ...
• 27.4k

### LightGBM - Why Exclusive Feature Bundling (EFB)?

I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of ...
• 490

### Does XGBoost handle multicollinearity by itself?

A remark on Sandeep's answer: Assuming 2 of your features are highly colinear (say equal 99% of time) Indeed only 1 feature is selected at each split, but for the next split, the xgb can select the ...

### Kappa near to 60% in unbalanced (1:10) data set

This may provide some answer: https://cran.r-project.org/web/packages/caret/vignettes/caret.html You may also check out Max Kuhn's "Applied Predictive Modeling" book. He talks about the caret package ...
• 1,057

### How to determine if my GBM model is overfitting?

I suggest training/testing your classifier on separate splits of the original dataset, and then printing a confusion matrix: https://topepo.github.io/caret/measuring-performance.html# This is a way ...
• 1,642

### Residuals in a gradient boosted classification

It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to ...
• 9,028

Welcome to the site! If I understand your question correctly you want to know why a model would perform worse when a new feature is added? So every time you do feature engineering (add new columns, ...
• 2,352
Accepted

### How does the number of trees effect the prediction time in gradient boost classification trees?

The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take ...
• 3,780