Ensemble Techniques - Boosting

Question

I understand boosting is a sequential learning technique and it use the prediction from previous model as a dataset for new model ,after adding weight to the misclassified data points. The point which was not clear how the weights are added for misclassified ones and diminished for the correctly classified ones. It would be great if veterans can help me to understand this

Thanks in advance

Answer 1

Actually, it depends on the boosting algorithm you used.

In the original boosting algorithm (Schapire 1990), three classifiers are used (say $C_1$, $C_2$ and $C_3$). The training dataset is randomly divided into three subsets, $D_1$, $D_2$ and $D_3$. The training process is as follow:

1. Train $C_1$ using $D_1$, and use $D_2$ to test $C_1$

2. The instances in $D_2$ which misclassified by $C_1$ (say m instances) are taken as training data for $C_2$ along with m instances that are correctly classified by $C_1$

3. Train $C_2$ using the 2*m instances taken from step 2

4. use $D_3$ to test both $C_1$ and $C_2$

5. Train $C_3$ on the instances that $C_1$ and $C_2$ disagree with each other (namely, the instances that $C_1$ and $C_2$ predicted differently)

---

For Adaboost, the training subsets are selected by sampling from the whole training set with probabilities.

At the beginning of running the algorithm, the probabilities are uniform. After a classifier is trained and tested, the probabilities of each successfully classified instances are decreased. Then, all the probabilities are normalized after adjustment. The next classifier is trained using dataset sampled with the adjusted probabilities.

reference: Ethem Alpaydin: Machine Learning