Hot answers tagged

60 votes
Accepted

GBM vs XGBOOST? Key differences?

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. ...
user avatar
  • 4,046
52 votes
Accepted

Does XGBoost handle multicollinearity by itself?

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models ...
user avatar
28 votes
Accepted

Why is xgboost so much faster than sklearn GradientBoostingClassifier?

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In ...
user avatar
  • 1,316
23 votes

Does XGBoost handle multicollinearity by itself?

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is ...
user avatar
22 votes

Does XGBoost handle multicollinearity by itself?

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, ...
user avatar
  • 330
18 votes

GBM vs XGBOOST? Key differences?

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in ...
user avatar
14 votes

GBM vs XGBOOST? Key differences?

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, ...
user avatar
  • 1,320
11 votes
Accepted

How fit pairwise ranking models in XGBoost?

According to the XGBoost documentation, XGboost expects: the examples of a same group to be consecutive examples, a list with the size of each group (which you can set with ...
user avatar
  • 246
9 votes
Accepted

Kappa near to 60% in unbalanced (1:10) data set

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing ...
user avatar
8 votes

Need help understanding xgboost's approximate split points proposal

I won't go into details but the following should help you grasp the idea. They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ ...
user avatar
  • 1,316
7 votes

Adding feature leads to worse results

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the ...
user avatar
  • 1,470
6 votes

What is init_score in lightGBM?

Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each ...
user avatar
  • 3,780
6 votes
Accepted

How to determine if my GBM model is overfitting?

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model ...
user avatar
6 votes

Need help understanding xgboost's approximate split points proposal

Just adding the algebraic part to @Winks answer: The second equation should have its sign reversed, as in: $$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - (-g_i/h_i)]^2 + constant = \sum_{i=1}^n\frac{1}{2}...
user avatar
  • 1,277
6 votes

difference between logistic regression and binary logistic regression

binary:logistic is used for binary classification where the target variable takes binary output [0, 1] reg:logistic is used for ...
user avatar
  • 509
6 votes

Can xgboost (or any other algorithm) give bad results with some bad features?

Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also ...
user avatar
6 votes

Decision Trees Nodes vs Leaves Definition

Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In ...
user avatar
  • 3,780
5 votes

How fit pairwise ranking models in XGBoost?

set_group is very important to ranking, because only the scores in one group are comparable. You can sort data according to their scores in their own group. For ...
user avatar
  • 51
5 votes
Accepted

Does the performance of GBM methods profit from feature scaling?

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that ...
user avatar
  • 5,664
5 votes
Accepted

Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the ...
user avatar
4 votes

Fit Decision Tree to Gradient Boosted Trees for Interpretability

Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual ...
user avatar
  • 961
4 votes
Accepted

Representing cyclical features as sin/cos components

as both components need to be equally weighted in order for the feature to make sense That is not the case. For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it ...
user avatar
  • 27.4k
4 votes

LightGBM - Why Exclusive Feature Bundling (EFB)?

I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of ...
user avatar
3 votes

Does XGBoost handle multicollinearity by itself?

A remark on Sandeep's answer: Assuming 2 of your features are highly colinear (say equal 99% of time) Indeed only 1 feature is selected at each split, but for the next split, the xgb can select the ...
user avatar
3 votes

Kappa near to 60% in unbalanced (1:10) data set

This may provide some answer: https://cran.r-project.org/web/packages/caret/vignettes/caret.html You may also check out Max Kuhn's "Applied Predictive Modeling" book. He talks about the caret package ...
user avatar
  • 1,057
3 votes

How to determine if my GBM model is overfitting?

I suggest training/testing your classifier on separate splits of the original dataset, and then printing a confusion matrix: https://topepo.github.io/caret/measuring-performance.html# This is a way ...
user avatar
  • 1,642
3 votes

Residuals in a gradient boosted classification

It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to ...
user avatar
3 votes

Adding feature leads to worse results

Welcome to the site! If I understand your question correctly you want to know why a model would perform worse when a new feature is added? So every time you do feature engineering (add new columns, ...
user avatar
  • 2,352
3 votes
Accepted

How does the number of trees effect the prediction time in gradient boost classification trees?

The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take ...
user avatar
  • 3,780
3 votes

Train classifier on balanced dataset and apply on imbalanced dataset?

Is this a problem? No. not at all. Will the model reproduce the trained target distribution from DS1 when applied on DS2? No, not necessarily. If Balanced set DS1 is a good representative of ...
user avatar
  • 8,629

Only top scored, non community-wiki answers of a minimum length are eligible