3
$\begingroup$

Suppose there are some classifiers as follows:

dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

In AdaBoost, I can define a base_estimator and also the number of estimators. However, I want to use these 7 classifiers. In other words, n_estimators=7 and these estimators are above ones. How can I define this model?

$\endgroup$
4
$\begingroup$

One possible solution is using a Stacking classifier as follows:

from sklearn.ensemble import StackingClassifier
# Assumed you already import  all the models you are using

dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

estimators = [
    ('rf', rf),
    ('svm1', svm1), ('svn2', svn2), ('xgb', xgb), ('knn', knn), ('lr', lr), ('dt', dt)
]

Option 1:

stacker = AdaBoostClassifier()   
model = StackingClassifier(
    estimators=estimators, final_estimator= stacker
)

Option 2:

base_model = StackingClassifier(
    estimators=estimators
)
model = AdaBoostClassifier(base_estimator = base_model)

Option 2 will be without question the most expensive* of both and as far as I understand, option 2 fits better what you are looking for.

  • Strategy:

enter image description here

$\endgroup$
3
  • $\begingroup$ Thank you, Julio. I have tried Stacking of these algorithms, but unfortunately, I do not know why it does not provide any improvement. Random Forest provides better results. $\endgroup$ – Katatonia Apr 6 at 23:28
  • $\begingroup$ You might be adding too much complexity (fitting noise) Why to use 2 different SVM and 3 different tree-based models? How correlated the predictions on each individual model are? Have you tried to tune the hyper parameters of individual and ensemble model? Have you tried to find the "optimal" subset of models according to your metric? $\endgroup$ – Julio Jesus Apr 7 at 1:57
  • $\begingroup$ Actually I tried some combinations but there is no improvement. I have not tested all combinations, but it seems that my data set is not completely identifiable. $\endgroup$ – Katatonia Apr 7 at 4:44
2
$\begingroup$

In practice, we never use any of the algorithms you list as base classifiers for Adaboost except of decision trees.

Adaboost (and similar ensemble methods) were conceived using decision trees (DTs) as base classifiers (more specifically, decision stumps, i.e. DTs with a depth of only 1); there is good reason why still today, if you don't specify explicitly the base_classifier argument in scikit-learn's AdaBoost implementation, it assumes a value of DecisionTreeClassifier(max_depth=1) (docs).

DTs are suitable for such ensembling because they are essentially unstable classifiers (this is also the reason they succeed as base classifiers in Random Forests, while you have never heard of "Random kNNs" or "Random SVMs"); this is not the case with SVM, kNN, or linear models, let alone models that they are themselves ensembles, like Random Forests and Boosted Trees (xgboost). Notice the following remark in the seminal paper by legendary statisticial (and RF inventor) Leo Breiman on Bagging Predictors:

Unstability was studied in Breiman [1994] where it was pointed out that neural nets, classifcation and regression trees, and subset selection in linear regression were unstable, while k-nearest neighbor methods were stable.

None of these algorithms (except Decision Trees) is expected to offer much when used as base classifiers for Adaboost (something you seem to have already discovered yourself, judging from the comments in the other answer). Attempting to use them simply because the framework (here scikit-learn) superficially allows us to do so is not a reason to do it.

See also the related Stack Overflow threads:

$\endgroup$
4
  • 1
    $\begingroup$ I found your point interesting, I just want to add, by unstable classifier we refer to one who has high variance such as dt and even nn. From my understanding the problem on the question's approach is that it seem to add "too much" complexity what makes really difficult for the model to lean no noise. So there is no ( I might be wrong) any theoretical reason not to use another base estimator (unstable one) for Adaboost. Am I missing something? $\endgroup$ – Julio Jesus Apr 8 at 17:18
  • 1
    $\begingroup$ Thanks for your comment. I did not consider these points. I have used RF, XGBoost, etc. to improve the performance but I could not. I have applied them in stacking model but there is no improvement. Could you please let me know what I should do if these algorithms do not improve the performance? What are the other options? $\endgroup$ – Katatonia Apr 8 at 20:10
  • $\begingroup$ @Katatonia I am afraid this is a completely different question, and it arguably cannot be resolved in a comment. Kindly consider accepting the answer, if it addressed the specific thing you asked here. $\endgroup$ – desertnaut Apr 8 at 20:15
  • 1
    $\begingroup$ @JulioJesus you are right about NNs (added citation), but have never heard of someone boosting NNs; about the rest, I am afraid we cannot say more without knowledge of OP's data. $\endgroup$ – desertnaut Apr 9 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.