3
$\begingroup$

I am learning boosting, the machine learning ensemble meta-algorithm. The professor is grouping 3 weak classifiers into an ensemble and said that before this time point it is easy to understand. Take a dataset, train a simple model, find the smallest error rate, something like this. This idea is easy to implement, for instance, gradient descent would take logistic regression to the smallest error rate. Then, the professor talked about the data with an exaggeration of classifier errors.

My question: What does that mean? Can anyone give a sample for this operation based on an open dataset in Python or R?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Not sure, but "exaggeration" could be an other way to talk about "overfitting".

Boosting are sequential model: each time you build a new tree, it will use the results of the precedent ones, and focus on residuals (where the precedent trees dont perform well). For this, the model exagerate the weights of the precedent mistakes.

If you build too much trees, the last ones will learn on noise, because residuals will contains much noise than information. You will have an excellent error rate ("exaggerated error rates") for your train dataset, but if you apply your model on a new dataset, the error will be lower.

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 for the second paragraph: in the video, he even mentions increasing weights for the misclassified points, which is the basis of AdaBoost. In gradient boosting, fitting the residuals is similar: "more wrongly" classified points contribute more to the loss function. That said, I disagree with your first sentence and last paragraph. While certainly you may find overfitting, it's not the same as the exaggeration the video discusses. $\endgroup$
    – Ben Reiniger ♦
    Commented Oct 8, 2019 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.