1
$\begingroup$

I am studying the Adaboost classification algorithm because i would like to implement it from scratch. I understand how it works, but i am not able to understand where some steps are placed.

I will describe the Adaboost training steps in my understanding (sorry for any incorrect formalism):

  • Initialize a weak learner $k$
  • Define a weight for each sample in the dataset equally $w =\frac{1}{N}$
  • Fit $k$ to the dataset
  • Calculate error $e = \sum_{i=0}^{N}e_iw_i$
  • Calculate importance $\alpha$ of $k$, i.e. $\alpha = \frac{1}{2}log(\frac{1-e}{e})$
  • Recalculate weights for the correct classified samples: $w_{t+1} = w e^{\alpha}$
  • Recalculate weights for the incorrect classified samples: $w_{t+1} = w e^{-\alpha}$
  • Normalize new sample weights: $w_{normalized} = \frac{w}{\sum_{i=0}^N w_i}$
  • For all sequent learners, select samples based in weighted random choice until get a dataset with same size as the original and do the same proccess.

My question is: how error is obtained? Regarding the implementation, should i firstly fit the dataset and then get the error from predicting the same dataset? This doesn't seems correct.

I've tried to read different sources about this and even a great explanation from the Statquest channel wasn't able to make this clear.

Thanks!

$\endgroup$
1
$\begingroup$

When I started learning about ensemble methods, this youtube video help me greatly.

The idea is that you fit the model on the data, calculate the error and then work with the error. So yes you should use a weak learner to predict your data and then, use another weak learner.

There are some more videos in the same channel that give a great explanation to have an intuition.

$\endgroup$
3
  • $\begingroup$ Oh ok, so this "error" is calculated by first: fitting the model to the data and using the same data to predict? Thanks for the video recommendation! $\endgroup$ Sep 9 '20 at 13:07
  • $\begingroup$ Yes, i reccomend you to see the Boosting video, it helped me greatly when coding from scratch $\endgroup$ Sep 9 '20 at 15:16
  • $\begingroup$ Sure, i will take a look at the video and then come back here to check your answer :D $\endgroup$ Sep 9 '20 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.