3 votes

Why does classifier (XGBoost) “after PCA” runtime increase compared to “before PCA”

This is likely behavior when several of your original features are discrete. Each tree, when splitting, considers a split for each unique value of each feature (in the current node). For discrete ...
Ben Reiniger's user avatar
  • 11.8k
3 votes

How to use a set of pre-defined classifiers in Adaboost?

One possible solution is using a Stacking classifier as follows: ...
Multivac's user avatar
  • 2,959
2 votes
Accepted

Is the way to combine weak learners in AdaBoost for regression arbitrary?

The ISL description is of gradient boosting (regression, with mse as the loss function), not of AdaBoost. There, $\lambda$ is constant, not weights for each tree. Since each tree is fitted to the ...
Ben Reiniger's user avatar
  • 11.8k
2 votes
Accepted

Scikit-learn's implementation of AdaBoost

The sklearn implementation of AdaBoost takes the base learner as an input parameter, with a decision tree as the default, so it cannot modify the tree-learning ...
Ben Reiniger's user avatar
  • 11.8k
2 votes

How to use a set of pre-defined classifiers in Adaboost?

In practice, we never use any of the algorithms you list as base classifiers for Adaboost except for decision trees. Adaboost (and similar ensemble methods) were conceived using decision trees (DTs) ...
desertnaut's user avatar
  • 1,988
1 vote
Accepted

How does Adaboost reassure us that It'll do better after each iteration?

Good question! You are correct that AdaBoost works by iteratively adding weak classifiers to the overall model to improve its accuracy. The key to understanding how AdaBoost ensures that each new ...
Pluviophile's user avatar
  • 3,838
1 vote

Evaluating optimal values for depth of tree

The training error shouldn't be too far from test error, otherwise it is a high deviance scenario and you could be in an overfitting situation in production. However, having a higher deviance could be ...
Nicolas Martin's user avatar
1 vote

How Adaboost calculates error for each weak learner in training?

When I started learning about ensemble methods, this youtube video help me greatly. The idea is that you fit the model on the data, calculate the error and then work with the error. So yes you should ...
Carlos Mougan's user avatar
1 vote
Accepted

Adaboost with other classifier fitting

Your description is apt. There isn't anything especially "mathematical" happening here, aside from the AdaBoost algorithm itself. In psuedocode, something like this is happening: ...
shadowtalker's user avatar
1 vote

Does gradient boosting algorithm error always decrease faster and lower on training data?

I'd like to confirm that this situation is not really something to worry about and I do not overfit the data. No, the situation is not worrying, you can consider it worrying when the test error ...
Carlos Mougan's user avatar
1 vote

Does gradient boosting algorithm error always decrease faster and lower on training data?

You should worry about overfitting when the test error rate starts to go up again. Until then I would set it aside. Overfitting is rather about the number of parameters, e.g. When two models with the ...
N. Kiefer's user avatar
  • 572
1 vote
Accepted

Why Adaboost SAMME needs f to be estimable?

Think for a while, if $f$ is not estimable, it can have any constant added to it with no difference on the result of the process. This means that if no other constraints are imposed, $f$ is not well/...
Nikos M.'s user avatar
  • 2,333
1 vote

Decreasing n_estimators is increasing accuracy in AdaBoost?

In short, AdaBoost works in that way that it trains in subsequent iterations and then measures the error of all available weak classifiers. In each subsequent iteration, the "validity" of incorrectly ...
fuwiak's user avatar
  • 1,373

Only top scored, non community-wiki answers of a minimum length are eligible