1
$\begingroup$

As far as i understand, weak learners of AdaBoost should never yield a error rate > 0.5

After training one, i only receive error rates above 0.5. How is that even possible? The AdaBoost Tree still gives quite good results, but all learners weights should be zero, so it should fail. Also the trees get worse from iteration to iteration

is it possible that my threshhold for the error rate instead is 0.9 (accuracy 0.1), as i got 10 classes and literature mostly focusses on binary cases?

from sklearn.ensemble import AdaBoostClassifier

adaboost_tree =  AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=max_d),
        n_estimators=estimators,
        learning_rate=1, algorithm='SAMME')
adaboost_tree.fit(data_train, labels_train)

enter image description here

$\endgroup$
2
$\begingroup$

As far as I understand, weak learners of AdaBoost should never yield a error rate > 0.5

This is only true for binary classification problems. The simplest possible classifier would pick the majority class, ensuring that accuracy is always >= 0.5 For a problem with $N$ classes, the worst error rate for a weak learner will be $\frac{N-1}{N}$.

Is it possible that my threshold for the error rate instead is 0.9 (accuracy 0.1), as I have 10 classes and literature mostly focuses on binary cases?

Yep that's exactly right. Your weak learners will have, at worst, an error rate of 0.9. Unsurprisingly, most of your learners have a better error rate better than random guessing, which explains the good performance of your ensemble.

Edit:

Also the trees get worse from iteration to iteration

This is a common feature in boosting algorithms. Each iteration, training samples are reweighted such that the mistakes from previous iterations are given more importance. The net effect is that the learners in later iterations are specialized for classifying the examples that earlier learners got wrong. Since later learners are specialized for a few hard-to-classify examples, they will have worse performance on the entire dataset.

$\endgroup$
  • 1
    $\begingroup$ thanks for the confirmation that makes sense to me as well. Do you have any idea why the error rate is strongly increasing though? $\endgroup$ – Quastiat Oct 14 '19 at 13:42
  • $\begingroup$ Oops, I forgot that part of your question! The worsening error rate for the learners in an emsemble is expected for boosting algorithms. I edited my answer with more details. $\endgroup$ – zachdj Oct 14 '19 at 13:54
  • 1
    $\begingroup$ thanks for the explanation as well, that sounds reasonable too. thanks a lot! $\endgroup$ – Quastiat Oct 14 '19 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.