Why might trees work so much better than boosting classifiers?

Question

I am predicting 10 classes label encoded using scikit-learn with 6 factors, 1.2M cases. DecisionTreeClassifier RandomForestClassifier ExtraTreesClassifier give accuracies (and precision and recall) of 0.9

AdaBoostClassifier GradientBoostingClassifier give accuracies of 0.2

Any pointers on the huge discrepancy?

(I am doing gridsearchcv). Code:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

def output_metrics():
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    print("Accuracy:",accuracy_score(y_test, y_pred))
    print('Precision', precision_score(y_test, y_pred, average=None).mean())
    print('Recall', recall_score(y_test, y_pred, average=None).mean())


from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV

tree_para =  { 'n_estimators': [16, 32] }
clf = GridSearchCV(AdaBoostClassifier(), tree_para, cv=5)
model= clf.fit(X_train, y_train)

y_pred = clf.predict(X_test) 

output_metrics()

Answer 1

As a disclaimer, I would like to point out that the performance in this specific case might still be due to the specific dataset used.

A likely explanation lies in the essence of what trees and boosting algorithms do.

As @akvall pointed out in the comments, Boosting algorithms can often overfit since this is what they are designed to do! As a reminder, regardless of how fancy a boosting algorithm works, it follows the following logic:

• train on the training set
• evaluate to see which mistakes were made
• retrain and focus more on the previous mistakes
• repeat until satisfactory results

Trees do not work the same way and are therefore less prone to overfitting. A Random Forest will simply compute independent trees and use majority voting to make a prediction.

Every boosting algorithm will "boost" for a certain amount of iterations, it is probably wise to look at how many times your algorithms did "boost".