40
votes
Accepted
Adaboost vs Gradient Boosting
Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion.
Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
oW_♦
- 6,264
11
votes
Accepted
What is a good interpretation of this 'learning curve' plot?
The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data.
The Y axis is an error score, so lower ...
- 25k
10
votes
Boosting with highly correlated features
Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables.
In general, tree based models are ...
- 469
9
votes
Accepted
Can Boosted Trees predict below the minimum value of the training label?
Yes, gradient boosted trees can make predictions outside the training labels' range. Here's a quick example:
...
- 11.2k
8
votes
Accepted
What is meant by Distributed for a gradient boosting library?
It means that it can be run on a distributed system (i.e. on multiple networked computers).
From XGBoost's documentation:
The same code runs on major distributed environment(Hadoop, SGE, MPI) and ...
- 340
8
votes
8
votes
Accepted
How to make LightGBM to suppress output?
As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here).
However, ...
- 1,567
7
votes
Accepted
Is the number of iterations in gradient tree boosting just the number of trees?
Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations.
From wiki:
One ...
- 4,103
7
votes
Accepted
How to extract trees in XGBoost?
This is an open feature request (at time of writing):
https://github.com/dmlc/xgboost/issues/2175
https://github.com/dmlc/xgboost/issues/3439
There, a very wasteful but working solution is mentioned: ...
- 11.2k
7
votes
GridSearch without CV
GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.
From the docs:
class sklearn....
- 578
6
votes
How to make LightGBM to suppress output?
To suppress (most) output from LightGBM, the following parameter can be set.
Suppress warnings: 'verbose': -1 must be specified in ...
- 7,366
5
votes
Accepted
which metric is better for boosting methods
Depends.
The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric.
Which one? Depends on which underlying problem you are ...
- 2,262
5
votes
Bagging vs Boosting, Bias vs Variance, Depth of trees
Question 1:
Bagging (Random Forest) is just an improvement on Decision Tree; Decision Tree has lot of nice properties, but it suffers from overfitting (high variance), by taking samples and ...
- 924
4
votes
Accepted
How are boosted decision stumps different from a decision tree?
The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner.
After step (1), a decision ...
oW_♦
- 6,264
4
votes
How to make LightGBM to suppress output?
Follow these points.
Use verbose= False in fit method.
Use verbose= -100 when you call the ...
4
votes
Accepted
How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?
I am going to talk about some ways you could do it later but first I want to talk about whether you should!
If the relation that you describe exists XGB will be able to learn and detect it! There is ...
- 1,723
4
votes
Accepted
XGBoost validation for number of trees
At first glance, your conclusion appears correct, but there are some important caveats to keep in mind.
First, what are the sizes of your training and validation sets? If your validation set is too ...
- 2,654
4
votes
XGBoost validation for number of trees
Just to add some general thoughts to the other answers. Gradient boosting is fairly robust to overfitting through increasing the number of trees. Increasing the number of trees is expected to increase ...
oW_♦
- 6,264
4
votes
Accepted
Extracting encoded features after CatBoost
I don't believe this is possible, CatBoost does target encoding per split, so you end up with different values of encoding at different trees.
Before each split is selected in the tree (see Choosing ...
- 924
4
votes
What is a good interpretation of this 'learning curve' plot?
It is pretty clear that your model is overfitting as your validation error is way higher than your training error.
This also means that more data allows your model to overfit less. If you are to have ...
- 6,031
4
votes
On gradient boosting and types of encodings
This is actually a feature of tree-based models in general, not just gradient boosting trees.
Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient.
On ...
- 2,654
3
votes
Accepted
Can parallel computing be utilized for boosting?
You can estimate in parallel each of the weak learners. For example, searching for optimal splits in 'weak' decision trees can be streamlined by utilizing large number of cores.
- 178
3
votes
Accepted
Does Gradient Boosting detect non-linear relationships?
GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all ...
- 531
3
votes
which metric is better for boosting methods
I would say AUC is the best overall metric for classification but does not have to be the only metric, accuracy is useful too. For reference you can check this Quora regarding accuracy vs. AUC:
...
- 4,103
3
votes
Accepted
Error rate of AdaBoost weak learner always bigger than 0.5?
As far as I understand, weak learners of AdaBoost should never yield a error rate > 0.5
This is only true for binary classification problems. The simplest possible classifier would pick the majority ...
- 2,654
3
votes
Accepted
Bagging vs Boosting, Bias vs Variance, Depth of trees
why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance)
Clearly it wouldn't make sense to bag a bunch of shallow trees/...
oW_♦
- 6,264
3
votes
GridSearch without CV
By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the ...
- 11.2k
3
votes
GridSearch without CV
Alternatively, just implement a simple Grid Search algorithm yourself. The book "Introduction to Machine Learning with Python" by Mueller and Guido includes an example using an ...
- 5,360
3
votes
Accepted
is it possible get a overfit underfit comparation between models, with this chart? (homework)
Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting ...
- 1,510
Only top scored, non community-wiki answers of a minimum length are eligible