1
$\begingroup$

I am currently testing out a few different ensemble methods on my dataset. I've heard that you can also use support vector machines as base learners in boosting and bagging methods but I am not sure which methods allow it or not. In particular, e.g. for XGB i tried out trees and SVMs as base learners and got the exact same result for 5 different performance metrics which made me question the results and/or that the option can only take trees as base learners. I didn't find much info in the documentation or at least not in all of the documentations. I would be interested about AdaBoostClassifier(), BaggingClassifier() and XGBClassifier(). Does anybody know the details and whether or not I can use SVMs here as base learners?

$\endgroup$
2
$\begingroup$

In short: Yes.

Conceptually, bagging and boosting are model-agnostic techniques, meaning that they work regardless of the learner.

Bagging essentially is the following:

  • create multiple predictors (they can even be hard-coded!)
  • gather predictions from the learners and come up with a prediction

Boosting can be seen as:

  • train a predictor
  • find where the predictor makes mistakes
  • put more emphasis on these mistakes
  • repeat until satisfactory

Regarding the specific Sklearn implementations, here are the base learners that you can use:

  • AdaBoostClassifier()

The documentation says Support for sample weighting is required, as well as proper classes_ and n_classes_ attributes.

This means that you can use all models that can give weight to your samples as part of the learning process (KNN, SVM, etc.)

  • BaggingClassifier()

This is a simple bagging strategy, so all estimators can be used here.

  • GradientBoostingClassifier()

This requires that your learners are differentiable so that the gradient can be computed. Generally, this technique is specific for tree learning.

$\endgroup$
3
  • 1
    $\begingroup$ This is true in theory, but the implementations may be different. XBG I think only supports trees (and linear models). Also, adaboost requires that the underlying model support instance weights (for the "put more emphasis" part). $\endgroup$ May 6 '20 at 1:01
  • 1
    $\begingroup$ Bagging and Boosting, as concepts, are model agnostic. XGBoost is a specific implementation of boosting which indeed requires trees as base learners. As for AdaBoost, if the model doesn't support instance weighting, you can still use a sampling strategy to mimic this (i.e. create your training set by sampling with replacement using the instance weights). I edited my answer to include your comment! $\endgroup$ May 6 '20 at 8:03
  • 1
    $\begingroup$ Perfect, this answers my question. Thank you! $\endgroup$
    – RazorLazor
    May 6 '20 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.