I am predicting 10 classes label encoded using scikit-learn with 6 factors, 1.2M cases. DecisionTreeClassifier RandomForestClassifier ExtraTreesClassifier give accuracies (and precision and recall) of 0.9

AdaBoostClassifier GradientBoostingClassifier give accuracies of 0.2

Any pointers on the huge discrepancy?

(I am doing gridsearchcv). Code:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

def output_metrics():
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    print("Accuracy:",accuracy_score(y_test, y_pred))
    print('Precision', precision_score(y_test, y_pred, average=None).mean())
    print('Recall', recall_score(y_test, y_pred, average=None).mean())

from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV

tree_para =  { 'n_estimators': [16, 32] }
clf = GridSearchCV(AdaBoostClassifier(), tree_para, cv=5)
model= clf.fit(X_train, y_train)

y_pred = clf.predict(X_test) 

  • 2
    $\begingroup$ It could be that boosting classifiers are overfitting. $\endgroup$
    – Akavall
    Dec 4, 2019 at 17:52
  • 1
    $\begingroup$ How do scores compare for train vs. validation/test data? $\endgroup$
    – Jonathan
    Dec 4, 2019 at 18:54
  • 2
    $\begingroup$ Hi, I would suggest you double-check these accuracies (e.g., calculate them yourself by hand). If I had to guess, I'd guess accuracy is being calculated differently for the different models. Perhaps 0.9 is one-vs-rest (2-way) accuracy and 0.2 is 10-way accuracy? What happens if you run scikit-learn.org/stable/modules/generated/… yourself for each best model? $\endgroup$
    – Robert
    Dec 4, 2019 at 19:23
  • $\begingroup$ @Akavall if they are overfitting wouldn't their metrics be better rather than worse than the trees? $\endgroup$
    – schoon
    Dec 5, 2019 at 9:58
  • 2
    $\begingroup$ I would like to see how well your AdaBoost learns the training data. You can get the accuracy of the best estimator in your GridSearchCVusing clf.best_score_. And clf.best_params_ would be interesting too. $\endgroup$
    – Jonathan
    Dec 5, 2019 at 11:04

1 Answer 1


As a disclaimer, I would like to point out that the performance in this specific case might still be due to the specific dataset used.

A likely explanation lies in the essence of what trees and boosting algorithms do.

As @akvall pointed out in the comments, Boosting algorithms can often overfit since this is what they are designed to do! As a reminder, regardless of how fancy a boosting algorithm works, it follows the following logic:

  • train on the training set
  • evaluate to see which mistakes were made
  • retrain and focus more on the previous mistakes
  • repeat until satisfactory results

Trees do not work the same way and are therefore less prone to overfitting. A Random Forest will simply compute independent trees and use majority voting to make a prediction.

Every boosting algorithm will "boost" for a certain amount of iterations, it is probably wise to look at how many times your algorithms did "boost".

  • $\begingroup$ If they are overfitting wouldn't their metrics be better rather than worse than the trees? $\endgroup$
    – schoon
    Dec 5, 2019 at 9:59
  • 1
    $\begingroup$ Neatly explained! $\endgroup$ Dec 5, 2019 at 10:58
  • $\begingroup$ @schoon if they overfit, their metrics will likely be better on the training set but worse on the test set, which seems to be what's happening now since GridSearchCV is used $\endgroup$ Dec 7, 2019 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.