38 votes
Accepted

Adaboost vs Gradient Boosting

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
user avatar
  • 5,995
11 votes
Accepted

What is a good interpretation of this 'learning curve' plot?

The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data. The Y axis is an error score, so lower ...
user avatar
  • 22.1k
9 votes

Boosting with highly correlated features

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables. In general, tree based models are ...
user avatar
  • 459
8 votes
Accepted

What is meant by Distributed for a gradient boosting library?

It means that it can be run on a distributed system (i.e. on multiple networked computers). From XGBoost's documentation: The same code runs on major distributed environment(Hadoop, SGE, MPI) and ...
user avatar
  • 340
8 votes
Accepted

Can Boosted Trees predict below the minimum value of the training label?

Yes, gradient boosted trees can make predictions outside the training labels' range. Here's a quick example: ...
user avatar
  • 9,698
7 votes
Accepted

How to make LightGBM to suppress output?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here). However, ...
user avatar
  • 1,517
7 votes
Accepted

How to extract trees in XGBoost?

This is an open feature request (at time of writing): https://github.com/dmlc/xgboost/issues/2175 https://github.com/dmlc/xgboost/issues/3439 There, a very wasteful but working solution is mentioned: ...
user avatar
  • 9,698
7 votes

GridSearch without CV

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds. From the docs: class sklearn....
user avatar
  • 568
6 votes

How to make LightGBM to suppress output?

To suppress (most) output from LightGBM, the following parameter can be set. Suppress warnings: 'verbose': -1 must be specified in ...
user avatar
  • 6,912
5 votes
Accepted

which metric is better for boosting methods

Depends. The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric. Which one? Depends on which underlying problem you are ...
user avatar
5 votes
Accepted

Is the number of iterations in gradient tree boosting just the number of trees?

Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations. From wiki: One ...
user avatar
5 votes

Bagging vs Boosting, Bias vs Variance, Depth of trees

Question 1: Bagging (Random Forest) is just an improvement on Decision Tree; Decision Tree has lot of nice properties, but it suffers from overfitting (high variance), by taking samples and ...
user avatar
  • 886
4 votes

How to make LightGBM to suppress output?

Solution for sklearn API (checked on v3.3.0): ...
user avatar
4 votes
Accepted

How are boosted decision stumps different from a decision tree?

The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner. After step (1), a decision ...
user avatar
  • 5,995
4 votes
Accepted

How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

I am going to talk about some ways you could do it later but first I want to talk about whether you should! If the relation that you describe exists XGB will be able to learn and detect it! There is ...
user avatar
  • 1,633
4 votes
Accepted

XGBoost validation for number of trees

At first glance, your conclusion appears correct, but there are some important caveats to keep in mind. First, what are the sizes of your training and validation sets? If your validation set is too ...
user avatar
  • 2,349
4 votes

XGBoost validation for number of trees

Just to add some general thoughts to the other answers. Gradient boosting is fairly robust to overfitting through increasing the number of trees. Increasing the number of trees is expected to increase ...
user avatar
  • 5,995
4 votes
Accepted

Extracting encoded features after CatBoost

I don't believe this is possible, CatBoost does target encoding per split, so you end up with different values of encoding at different trees. Before each split is selected in the tree (see Choosing ...
user avatar
  • 886
4 votes

What is a good interpretation of this 'learning curve' plot?

It is pretty clear that your model is overfitting as your validation error is way higher than your training error. This also means that more data allows your model to overfit less. If you are to have ...
user avatar
  • 5,664
3 votes
Accepted

Does Gradient Boosting detect non-linear relationships?

GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all ...
user avatar
  • 531
3 votes

which metric is better for boosting methods

I would say AUC is the best overall metric for classification but does not have to be the only metric, accuracy is useful too. For reference you can check this Quora regarding accuracy vs. AUC: ...
user avatar
3 votes
Accepted

Error rate of AdaBoost weak learner always bigger than 0.5?

As far as I understand, weak learners of AdaBoost should never yield a error rate > 0.5 This is only true for binary classification problems. The simplest possible classifier would pick the majority ...
user avatar
  • 2,349
3 votes
Accepted

Bagging vs Boosting, Bias vs Variance, Depth of trees

why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance) Clearly it wouldn't make sense to bag a bunch of shallow trees/...
user avatar
  • 5,995
3 votes

GridSearch without CV

By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the ...
user avatar
  • 9,698
3 votes
Accepted

is it possible get a overfit underfit comparation between models, with this chart? (homework)

Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting ...
user avatar
3 votes

On gradient boosting and types of encodings

This is actually a feature of tree-based models in general, not just gradient boosting trees. Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient. On ...
user avatar
  • 2,349
3 votes

Which other algorithms fit residuals like XGBoost?

Training of XGBoost is based on a boosting model, which is a general ensemble method creating a strong model from a number of weak models. This process is performed by building a model from the ...
user avatar
  • 648
3 votes

Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”

This is likely behavior when several of your original features are discrete. Each tree, when splitting, considers a split for each unique value of each feature (in the current node). For discrete ...
user avatar
  • 9,698
2 votes
Accepted

Is ensemble learning using different classifier combination another name for Boosting?

Boosting is a type of Ensemble Learning, but it is not the only one. Apart from stacking, bagging is also another type of Ensemble Learning. Ensemble Learning is the combination of individual models ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible