5
$\begingroup$

I am trying to implement the AdaBoost.M1 algorithm (trees as base-learners) to a data set with a large feature space (~ 20.000 features) and ~ 100 samples in R. There exists a variety of different packages for this purpose; adabag, ada and gbm. gbm() (from the gbm-package) appears to be my only available option, as stack.overflow is a problem in the others, and though it works, it is very time-consuming.

Questions:

  1. Is there any way to overcome the stack.overflow the problem in the other packages, or have the gbm() run faster? I have tried converting the data.frame into a matrix without success.
  2. When performing AdaBoost in gbm() (with distribution set to "adaboost"), An Introduction to Statistical Learning (Hastie et al.) mentions the following parameters needed for tuning:

Needs:

  1. The total number of trees to fit.
  2. The shrinkage parameter denoted lambda.
  3. The number of splits in each tree, controlling the complexity of the boosted ensemble.

As the algorithm is very time consuming to perform in R, I need to find literature on what tuning parameters that are within a suitable range for this kind of large feature space data, before performing cross-validation over the range to estimate the test error rate.

Any suggestions?

|improve this question
$\endgroup$
  • 1
    $\begingroup$ Is there any reason you have to use adaboost? gradient boosting (xgboost), as I know it, is more flexible and customization than adaboost and certainly faster in R. Here is an excellent answer on how to hypertune a gradient booster using the caret package: stats.stackexchange.com/questions/171043/… $\endgroup$ – data science guy Aug 18 '17 at 14:48
  • 1
    $\begingroup$ I believe that for such case with much more features then samples no method would work well, you should consider some feature space reduction with for example PCA $\endgroup$ – Jirka B. Apr 23 '19 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.