Hot answers tagged

10 votes

Boosting with highly correlated features

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables. In general, tree based models are ...
Michael M's user avatar
  • 479
4 votes

Bagging vs pasting in ensemble learning

Let's say we have a set of 40 numbers from 1 to 40. We have to pick 4 subsets of 10 numbers. Case 1 - Bagging - We will pick the first number, put it back, and then pick the next. This makes all the ...
10xAI's user avatar
  • 5,574
2 votes
Accepted

Can I do bagging method as improvement technique to decision tree in research?

Let's clarify a few things first: The bagging technique is an ensemble method which is not specific to decision trees, it can be applied to any classification method. It's worth noting that there is ...
Erwan's user avatar
  • 25.2k
2 votes
Accepted

Can Boosting and Bagging be applied to heterogeneous algorithms?

The short answer is yes. Both boosting and bagging meta-algorithms do not assume specific weak learners, thus any learner can do, no matter if uses same algorithm or different one. The way the meta-...
Nikos M.'s user avatar
  • 2,301
2 votes
Accepted

Can the product of tree regressions be represented by a single tree?

Yes, it is possible to find a tree that represents the product. An easy way to do so is to extend the first tree in each leaf with the second tree. Example Assume there are theses two tree: A ...
Broele's user avatar
  • 1,352
2 votes
Accepted

Why the accuracy of my bagging model heavily affected by random state?

Can someone explain why the huge gap It simply means that there's a quite high variance depending which random set of instances is picked. How many times do you re-sample the instances in the bagging ...
Erwan's user avatar
  • 25.2k
2 votes
Accepted

Random Forest Stacking Experiment for Imbalanced Data-set Problem

The underlying idea is fine, but you've fallen into a common data leakage trap. By recombining the data and then resplitting, your second model's test set includes some of the first model's training ...
Ben Reiniger's user avatar
  • 11.6k
1 vote

How a Random forest "learns" or How loss (objective function value) is propagated back so that a random forest can "Improve"?

In a random forest classifier, there is no backpropagated loss. Instead, the N trees are grown independently from each other and then, for a new prediction, a ...
justinlk's user avatar
  • 136
1 vote
Accepted

Counting the number of trainable parameters in a gradient boosted tree

Maybe that can be the size of your grid, but not the number of possible trainable hyperparameter of the Gradient Boosting Regressor of scikit learn. I add some more hyperparameters of GBDT Regressor: ...
Carlos Mougan's user avatar
1 vote

What are valid measures for reporting k-fold score in the case of confusion-matrix?

You can just sum all the cells across folds: for every true class $T_i$ and every predicted $P_j$, the number is the sum of this cell $(T_i,P_j)$ for fold 1, fold 2, .., fold N. This is because the ...
Erwan's user avatar
  • 25.2k
1 vote

Where should I stop training if I want to bag models

You should not compare your machine learning task with others, particularly when they are overfitting their models (on other tasks). Second, there is no mathematical rule for fixing the number $k$ of ...
Eduard's user avatar
  • 659
1 vote

Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

No, the samples will not be independent, there is possibility the data samples will be skewed. For example, imagine a class-imbalanced binary problem, once the minority class is already sampled (large ...
Nikos M.'s user avatar
  • 2,301
1 vote
Accepted

Difference between bagging and pasting?

When a sampling unit is drawn from a finite population and is returned to that population, after its characteristic(s) have been recorded, before the next unit is drawn, the sampling is said to be “...
Peter's user avatar
  • 7,426
1 vote

Base model in ensemble learning

Intuitively speaking, ensembles benefit most from diversity. Imagine being in a room of people making a decision together. If everyone more or less agrees, you don't benefit from having more people at ...
Valentin Calomme's user avatar
1 vote

bagging vs. pasting in ensemble learning

Actually I think that you are mostly correct, except that in my understanding "with/without replacement" applies only to selecting one subset, not across subsets. This means that if we have ...
Erwan's user avatar
  • 25.2k
1 vote

Bagging Base models

Bagging, also called bootstrap aggregation, reduces overfitting by considering an ensemble of weak learners. This doesnot mean that the model underfits. The weak classifiers or regressors(usually ...
Anoop A Nair's user avatar
1 vote

How does bagging help reduce the variance

High Variance - Model varies a lot on small changes High Bias - Model doesn't vary so much but predict quite away from the truth Let's check a Decision Tree on 5 values - \begin{array} {|r|r|} \...
10xAI's user avatar
  • 5,574

Only top scored, non community-wiki answers of a minimum length are eligible