Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in all the iterations, could we use different algorithms?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The short answer is yes. Both boosting and bagging meta-algorithms do not assume specific weak learners, thus any learner can do, no matter if uses same algorithm or different one.
The way the meta-algorithms are defined, they use the weak learners as black-box models, without reference to their implementation or algorithmic principle, nor similarity.
For Boosting:
In machine learning, boosting is an ensemble meta-algorithm for primarily reducing bias, and also variance[1] in supervised learning, and a family of machine learning algorithms that convert weak learners to strong ones.[2] Boosting is based on the question posed by Kearns and Valiant (1988, 1989):[3][4] "Can a set of weak learners create a single strong learner?" A weak learner is defined to be a classifier that is only slightly correlated with the true classification (it can label examples better than random guessing). In contrast, a strong learner is a classifier that is arbitrarily well-correlated with the true classification.
For Bagging:
Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach.
For Bagging the required condition to improve performance is that the weak learners should be instable (thus perturbed versions of the learners affect outcomes), but other than that, learners are black boxes, as mentioned above.
For further reference:
