Hot answers tagged

56

Quote from the author of xgboost: Both xgboost and gbm follows the principle of gradient boosting. There are however, the difference in modeling details. Specifically, xgboost used a more regularized model formalization to control over-fitting, which gives it better performance. We have updated a comprehensive tutorial on introduction to the model, which ...


45

Decision trees are by nature immune to multi-collinearity. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. Since boosted trees use individual decision trees, they also are unaffected by multi-collinearity. ...


27

Since you mention "numeric" features, I guess your features are not categorical and have a high arity (they can take a lot of different values, and thus there are a lot of possible split points). In such a case, growing trees is difficult since there are [a lot of features $\times$ a lot of split points] to evaluate. My guess is that the biggest effect ...


18

There is an answer from Tianqi Chen (2018). This difference has an impact on a corner case in feature importance analysis: the correlated features. Imagine two features perfectly correlated, feature A and feature B. For one specific tree, if the algorithm needs one of them, it will choose randomly (true in both boosting and Random Forests™). ...


18

I was curious about this and made a few tests. I’ve trained a model on the diamonds dataset, and observed that the variable “x” is the most important to predict whether the price of a diamond is higher than a certain threshold. Then, I’ve added multiple columns highly correlated to x, ran the same model, and observed the same values. It seems that when ...


17

In addition to the answer given by Icyblade, the developers of xgboost have made a number of important performance enhancements to different parts of the implementation which make a big difference in speed and memory utilization: Use of sparse matrices with sparsity aware algorithms Improved data structures for better processor cache utilization which makes ...


14

One very important difference is xgboost has implemented DART, the dropout regularization for regression trees. References Rashmi, K. V., & Gilad-Bachrach, R. (2015). Dart: Dropouts meet multiple additive regression trees. arXiv preprint arXiv:1505.01866.


11

According to the XGBoost documentation, XGboost expects: the examples of a same group to be consecutive examples, a list with the size of each group (which you can set with set_group method of DMatrix in Python).


9

The Kappa is Cohen's Kappa score for inter-rater agreement. It's a commonly-used metric for evaluating the performance of machine learning algorithms and human annotaters, particularly when dealing with text/linguistics. What it does is compare the level of agreement between the output of the (human or algorithmic) annotater and the ground truth labels, to ...


8

I won't go into details but the following should help you grasp the idea. They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ (sorted), you can try the $10$-quantiles split points $\{x_{10}, x_{20}, \cdots, x_{90}\}$ and have a good approximation already. This is what the $\epsilon$ ...


7

To put it shortly, xgboost tries to fix it and although it is very good in getting rid of overfitting, it is not perfect. Adding new features is not always beneficial, because you increase the dimension of your search space and thus make the problem harder. In your particular case the increased complexity overweight the added value from extra features. ...


6

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data. Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting. To see if you are overfitting, split your ...


6

Just adding the algebraic part to @Winks answer: The second equation should have its sign reversed, as in: $$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - (-g_i/h_i)]^2 + constant = \sum_{i=1}^n\frac{1}{2}h_i[f_t^2(x_i) + 2\frac{f_t(x_i)g_i}{h_i} + (g_i/h_i)^2] = \sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i) + \frac{gi^2}{2h_i}]$$ The last term is indeed ...


6

Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of ...


6

binary:logistic is used for binary classification where the target variable takes binary output [0, 1] reg:logistic is used for regression where the target variable is continuous between [0, 1] Quote from xgboost doc: We use linear regression here, if we want use objective = reg:logistic logistic regression, the label needed to be pre-scaled into [0,1] ...


6

Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if ...


6

Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In your example tree diagram, the nodes that say 'Large', 'Medium' or 'Small' are leaf nodes. The other nodes in the tree are interchangeably called split nodes, ...


5

set_group is very important to ranking, because only the scores in one group are comparable. You can sort data according to their scores in their own group. For easy ranking, you can use my xgboostExtension.


5

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions ...


4

Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual produced by the predictions from the earlier trees. The explain method shows for each prediction (i.e. record) why a particular decision was made. This results in a ...


4

as both components need to be equally weighted in order for the feature to make sense That is not the case. For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it means that the original feature has the strongest positive effect on output at $\theta = \frac{\pi}{2}$. If the two features are weighted equally, then the focus ...


4

I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of their intuitive idea of bundling of the features in an efficient way. But specificly, it does not say that it does a 'reversion of one-hot-encoding' in particular ...


4

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent. Given this, you can boost any kind of model that you want (as far as I know)....


3

I suggest training/testing your classifier on separate splits of the original dataset, and then printing a confusion matrix: https://topepo.github.io/caret/measuring-performance.html# This is a way of seeing how many of the 'true' classifications your classifier predicted correctly or incorrectly, and the same for 'false' classifications. This will give you ...


3

This may provide some answer: https://cran.r-project.org/web/packages/caret/vignettes/caret.html You may also check out Max Kuhn's "Applied Predictive Modeling" book. He talks about the caret package at length in this book, including the kappa statistics and how to use it. This may be of some help to you.


3

It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to a probability. The loss function used for deciding the weights of the terminal nodes is adapted from the normal sigmoid loss to not have to map directly to ...


3

Welcome to the site! If I understand your question correctly you want to know why a model would perform worse when a new feature is added? So every time you do feature engineering (add new columns, derive columns, standardize the data, normalize the data, etc) there is always a flip side of the coin. If you add some features and if those features explain ...


3

The time it takes to get a prediction from a model of gradient boosted classification trees should be linear in the number of trees. So getting predictions from a model with 1000 trees should take about twice as long as 500 trees, and about half as long as 2000 trees. You'll need to test it yourself and check if it's fast enough for your use case. Modern ...


3

Is this a problem? No. not at all. Will the model reproduce the trained target distribution from DS1 when applied on DS2? No, not necessarily. If Balanced set DS1 is a good representative of imbalanced (target) set DS2, and Classes are well-separated (pointed out by @BenReiniger), which holds easier in higher dimensions, then model will generate ...


2

I am hesitant to think of features as bad features. However, there are features more or less useful relative to other features. In general pruning features are seen as a good best practice. Overfitting certainly is a real problem within machine learning. Usually a combination of creating too powerful of a model, not having enough data, and/or having too ...


Only top voted, non community-wiki answers of a minimum length are eligible