Can all types of ML methods benefit from bagging? Decision Tree Classification seems always be the go-to example of bagging, what about other classifiers or regressions?

When it's suitable to do bagging, how to pick the size and number of bags?


I am looking for something mathematically more rigorous, such as, for each model (single learner) we can break its total estimation error into:

$Error^2 = Bias^2 + Variance^2 + Irreducible^2$

If we can have a rough estimation of $Variance$ and the correlation between the predictions from all the single learners, then we can know about how low we can push variance to through ensemble.


Bagging main goal is to minimize variance of your model. Basically, if you have a model that is on average pretty accurate but inconsistent (meaning, it does well for a given data set, poorly generalizations) then bagging may be a way to produce a more consistent estimators. Decision trees are the common example of this because they are the canonical high variance machine learning algorithm.

As for your last question, the size of each new training set needs to be the same size as the original training set. The way you achieve this is by random sampling of the original dataset with replacement (meaning the new dataset may have duplicates). The number of new training sets is dependent on the problem. Sometimes 100 is fine other times you need 1000 or so. There isn't a way to just know how many sets you need. It is a parameter that needs to be tuned.

  • $\begingroup$ Thanks! What other models do we usually apply bagging on? I am actually looking for whether there is any way to quantify the reduction of variance, say for a particular model, if we can roughly estimate the 3 components that total estimation error consists of, namely bias, variance and irreducible noise, then we can strike a balance between number of bags and the reduction in variance. $\endgroup$ – Indominus Jan 14 '19 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.