0
$\begingroup$

For implementation I am following the Matlab code for AdaBoost. Based on my understanding, AdaBoost uses weak classifiers known as base classifiers and creates several instances of it. For example, a weak classifier is a decision tree. So, AdaBoost can create maximum N decision trees (where N = number of samples) and combine the prediction results. This is a Homogeneous boosting method. But I have seen some examples such as this one in Matlab and ensemble-toolbox which have confused me. Can somebody please explain the following concepts with respect to the implementations and what is going on in the code?

1) Does the Matlab code for AdaBoost combine different classifiers? The combination method is unclear to me - -whether they do sum or majority voting or something else.

If they are combining several classifiers, then technically it is a Heterogenous ensemble method and the term for it is stacking and not boosting. Please correct me where I am wrong. In Boosting methods, the based classifiers are the same. But the given in Matlab code for AdaBoost combines different classifiers, I am not sure.

2) Is ensemble learning or the example in the ensemble toolbox the same as the Adaptive Boosting Matlab code (second link)? Is ensemble learning the same as Adaptive boost?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Boosting is a type of Ensemble Learning, but it is not the only one. Apart from stacking, bagging is also another type of Ensemble Learning.

Ensemble Learning is the combination of individual models together trying to obtain better predictive performance that could be obtained from any of the constituent learning algorithms alone.

Boosting involves incrementally building an ensemble by training each new model instance to emphasize the training instances that previous models mis-classified. It is an iterative technique which adjust the weight of an observation based on the last classification. If an observation was classified incorrectly, it tries to increase the weight of this observation and vice versa. Boosting in general decreases the bias error and builds strong predictive models. Sometimes they may over fit on the training data.

Stacking involves training a learning algorithm to combine the predictions of several other learning algorithms.

Bagging tries to implement similar learners on small sample populations and then takes a mean of all the predictions. In generalized bagging, you can use different learners on different population. As you can expect this helps us to reduce the variance error.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer as well. If I have followed your explanation correctly then the Matlab example in the first link of my question uses different classifiers such as svm, k-nn, linear regression etc and combines the hypothesis by weights. Such a type falls under stacking as different classifiers are used. However, the Author calls it Adaptive boosting. This is the source of confusion. It would be kind of you to please clarify this. $\endgroup$
    – Srishti M
    Jun 20, 2018 at 22:52
  • $\begingroup$ Actually stacking is similar to boosting, because you can also apply several models to your original data. But there is one difference between them. In stacking approach, you don't just have an empirical formula for your weight function, rather you introduce a meta-level and use another model to estimate the input together with outputs of every model in order to determine what models perform well given this input data set. $\endgroup$
    – Christos Karatsalos
    Jun 20, 2018 at 23:22
  • $\begingroup$ I see, to summarize from our discussions, boosting can be performed using one kind of classifiers as well as a collection of different kinds of base classifiers with a well-defined formula which is the weighted combination of the hypothesis. Whereas, in stacking it is upto the programmer to come up with a method to combine the decision of the different classifiers. On the other hand, ensemble learning using different classifiers can be performed by applying hypothesis combination methods such as majority vote, sum, maximum, dempster method etc. Is my understanding correct? $\endgroup$
    – Srishti M
    Jun 21, 2018 at 0:15
  • $\begingroup$ Yes. For the last part, all types bagging, boosting and stacking are supposed to be types of Ensemble Learning and as you can see Ensemble Learning includes a more generic definition above. $\endgroup$
    – Christos Karatsalos
    Jun 21, 2018 at 7:46
2
$\begingroup$

Ensemble learning combines predictions from multiple learners. Boosting methods are one way to form an ensemble. Stacking is another. The important difference between boosting and stacking (and other ensemble methods) is that boosting applies a number of weak learners sequentially and then produces a final result via a weighted majority vote.

The learners in stacking can also be combined as a weighted average or vote (or by another "meta" model) but they can be more or less independent.

In boosting, each weak learner (usually the same, yes) modifies the data for the next learner in the sequence. E.g. in AdaBoost the weight of each learner and also each sample in the data depends on the misclassification rate of the previous weak learner to the effect that the next learner focuses more on previously misclassified samples.

Boosting methods typically vary by how these weight updates are performed.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you for your answer. Based on your answer, both the implementations are doing doing boosting and then combining the results of each of the different classifier to get another classifier: sign(Hypothesis*weight_hypothesis). Is the last step stacking since the hypothesis from independent models are used?(variable Hypothesis). Could you please clarify this? $\endgroup$
    – Srishti M
    Jun 20, 2018 at 18:56
  • $\begingroup$ Adaptive boosting can be done combining different weak classifiers Or same classifier or both? In these two implementations, there is step where all the different classifiers are combined as well, I think. I am not sure what is going on. $\endgroup$
    – Srishti M
    Jun 20, 2018 at 19:06
  • $\begingroup$ what exactly are you referring to? $\endgroup$
    – oW_ ♦
    Jun 20, 2018 at 20:02
  • $\begingroup$ Perhaps I was not clear in my comments. I will reword it.(1) Can AdaptiveBoosting be done by combining different classifiers (then it is adaptive stacking) and also by using the same classifier and combining several instances of it? (2) In my question I have given 2 links for the Matlab implementation. I guess that both the codes are doing adaptive boosting by combining different classifiers. If they are combining several classifiers, then technically it is a Heterogenous ensemble method and the term for it is stacking and not boosting. Please correct me where I am wrong. $\endgroup$
    – Srishti M
    Jun 20, 2018 at 20:33
  • $\begingroup$ AdaBoost can be (but usually is not) performed with different models. This is what the first example does, even though in my opinion the implementation doesn't make a whole lot of sense. The second example just provides a few options for combining different classifier, e.g. majority vote, taking the max etc. none of which is boosting. $\endgroup$
    – oW_ ♦
    Jun 20, 2018 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.