2
$\begingroup$

I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset. (X,Y are the predictor columns and the color is the label)

enter image description here

As you can see there are exactly 16 points in either side that can be easily miss-classified. TO check how the performance increases with increasing n_estimators I used this code

for i in range(1,21):
    clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2),
            n_estimators=i,algorithm='SAMME')
    clf.fit(X_df,y)
    y_pred = clf.predict(X_df)

    from sklearn.metrics import confusion_matrix as CM
    #CM(y_pred,y_pred1)
    #CM(y,y_pred1)
    print(i,CM(y,y_pred))

Upto n_estimators = 13 all the 32 points were miss classified. The confusion matrix is

[[84 16]
 [16 84]]

( with the exception of n_estimators=8. Here all red points were properly classified)

[[100 0]
 [16 84]]

From 13 onwards it began flipping in a weird way. The confusion matrices are given in order.

[[84 16] | [[ 84  16] | [[84 16] | [[ 84  16] | [[ 84  16] | [[100   0] | [[100   0] | [[84 16]
[16 84]] | [  0 100]] | [16 84]] | [  0 100]] | [  0 100]] | [ 16  84]] | [  0 100]] | [16 84]]

Where apparently n_estimators=19 is giving better performance than n_estimators = 20.

Can someone please explain what is happening and what is causing this behavior?

$\endgroup$
1
$\begingroup$

In short, AdaBoost works in that way that it trains in subsequent iterations and then measures the error of all available weak classifiers. In each subsequent iteration, the "validity" of incorrectly qualified observations is increased, so that classifiers pay more attention to them. So confusion matrix could be shown after each iteration(after 13). In case n_estimators=19 its look like you have the perfect fit so for bigger values of n_estimators model starts to overfit, which gives worse performance. In your case please read about early stopping. That technique help you find you best value of n_estimators.

https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_early_stopping.html

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.