Boosting is a sequential technique in which, the first algorithm is trained on the entire dataset and the subsequent algorithms are built by fitting the residuals of the first algorithm, thus giving higher weight to those observations that were poorly predicted by the previous model. Examples are adaboost and GBM, etc. My question is that how to perform boosting when ensembling the base learner? Especially how to get the residual if it is a classification problem?

I know how to bagging the base learners and stacking the base learners. I just have no idea how to boosting the base learners.



2 Answers 2


You can use any base learner for boosting (Adaboost requires sample weighting though). Keep in mind however that the original idea is to use weak learners with strong bias and reducing that bias through boosting.

If it is a classification problem, usually logarithmic loss is used to calculating the residual/gradient for boosting.

For Python, there is a nice AdaBoost wrapper in scikit-learn (AdaBoostClassifier) which can take for example a Random Forest as base learner.


To extend to Alex's answer, boosting learning has several distinct features, compared to bagging,

  1. It is performed in a sequential steps.

  2. At each step, it uses the residual rather than label to learn. The residual function for classification could be logloss, for regression could be MSE.

  3. At each step, no sampling is required so it can use entire dataset

  4. each step is a weak learner, which means that for tree based algorithms, it uses tree stumps that has only one split, vs bagging, each model are fully grown.

  5. when learning from residuals from previous steps, the residuals/problems are magnified so that it helps training in this step. This is done using weights. The improvement made by previous model is multiplied by a factor alpha, which is typically 0.01. ( details referring to ISL book )

happy learning!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.