1
$\begingroup$

I am predicting 10 classes label encoded using scikit-learn with 6 factors, 1.2M cases. DecisionTreeClassifier RandomForestClassifier ExtraTreesClassifier give accuracies (and precision and recall) of 0.9

AdaBoostClassifier GradientBoostingClassifier give accuracies of 0.2

Any pointers on the huge discrepancy?

(I am doing gridsearchcv). Code:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

def output_metrics():
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    print("Accuracy:",accuracy_score(y_test, y_pred))
    print('Precision', precision_score(y_test, y_pred, average=None).mean())
    print('Recall', recall_score(y_test, y_pred, average=None).mean())


from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV

tree_para =  { 'n_estimators': [16, 32] }
clf = GridSearchCV(AdaBoostClassifier(), tree_para, cv=5)
model= clf.fit(X_train, y_train)

y_pred = clf.predict(X_test) 

output_metrics()
$\endgroup$
  • 2
    $\begingroup$ It could be that boosting classifiers are overfitting. $\endgroup$ – Akavall Dec 4 '19 at 17:52
  • 1
    $\begingroup$ How do scores compare for train vs. validation/test data? $\endgroup$ – Sammy Dec 4 '19 at 18:54
  • 2
    $\begingroup$ Hi, I would suggest you double-check these accuracies (e.g., calculate them yourself by hand). If I had to guess, I'd guess accuracy is being calculated differently for the different models. Perhaps 0.9 is one-vs-rest (2-way) accuracy and 0.2 is 10-way accuracy? What happens if you run scikit-learn.org/stable/modules/generated/… yourself for each best model? $\endgroup$ – Robert Dec 4 '19 at 19:23
  • $\begingroup$ @Akavall if they are overfitting wouldn't their metrics be better rather than worse than the trees? $\endgroup$ – schoon Dec 5 '19 at 9:58
  • 2
    $\begingroup$ I would like to see how well your AdaBoost learns the training data. You can get the accuracy of the best estimator in your GridSearchCVusing clf.best_score_. And clf.best_params_ would be interesting too. $\endgroup$ – Sammy Dec 5 '19 at 11:04
2
$\begingroup$

As a disclaimer, I would like to point out that the performance in this specific case might still be due to the specific dataset used.

A likely explanation lies in the essence of what trees and boosting algorithms do.

As @akvall pointed out in the comments, Boosting algorithms can often overfit since this is what they are designed to do! As a reminder, regardless of how fancy a boosting algorithm works, it follows the following logic:

  • train on the training set
  • evaluate to see which mistakes were made
  • retrain and focus more on the previous mistakes
  • repeat until satisfactory results

Trees do not work the same way and are therefore less prone to overfitting. A Random Forest will simply compute independent trees and use majority voting to make a prediction.

Every boosting algorithm will "boost" for a certain amount of iterations, it is probably wise to look at how many times your algorithms did "boost".

$\endgroup$
  • $\begingroup$ If they are overfitting wouldn't their metrics be better rather than worse than the trees? $\endgroup$ – schoon Dec 5 '19 at 9:59
  • 1
    $\begingroup$ Neatly explained! $\endgroup$ – IamTheRealFord Dec 5 '19 at 10:58
  • $\begingroup$ @schoon if they overfit, their metrics will likely be better on the training set but worse on the test set, which seems to be what's happening now since GridSearchCV is used $\endgroup$ – Valentin Calomme Dec 7 '19 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.