Currently i have the following code

xgb_model = Pipeline([
    ('scaler', StandardScaler()),
    ('imputer', SimpleImputer(strategy='median')),
    ('xgb', XGBClassifier(subsample=1, min_child_weight=10, max_depth=3, gamma=1.5, colsample_bytree=1))

model6 = Pipeline([
    ('scaler', RobustScaler()),
    ('imputer', KNNImputer()),
    ('RF', RandomForestClassifier(n_estimators=200, min_samples_split=12, min_samples_leaf=5,
                                  max_features=2, max_depth=80, bootstrap=True, n_jobs=-1))
estimators = [
    ('xgb', xgb_model),  
    ('model6', model6)   

ensemble = VotingClassifier(estimators, voting='soft', verbose=10)

ensemble.fit(train.drop(['Unnamed: 0', 'SeriousDlqin2yrs'], axis=1), train['SeriousDlqin2yrs'])
predictions = ensemble.predict_proba(test.drop(['Unnamed: 0', 'SeriousDlqin2yrs'], axis=1))

But i want to add another model to the ensemble. Can i do that without having to re-run the existing code to save some time? I don't know if it's theoretically right to kind of do a new_ensemble between ensemble and model7.

If Voting get each model's probability separately, can't i fit another model and add it to the previous ensemble? If it's not wrong, is doing it manually the only way out?


1 Answer 1


[Edited as part of the answer wasn't correct/precise]

The only part that needs re-fit is the estimators and after. You do not have to retrain xgb_model and model6.

If you are on Jupyter notebook just run them in separate cells. Otherwise on Python script you can save the models first, and load them when needed.

In your case, running ensemble.fit() will train the whole pipeline starting from base models xgb_model and model6 onward; I haven't found a way to 'freeze' them, add an untrained model, then fit it and the VotingClassifier only. If anyone has an idea, please kindly share.

So the next best thing you can do is to train the ensemble once, and take its predictions to fuse with other models. Or you can take out the individual predictions of each of the base models via syntax <cls>.named_estimators_.<name of estimator>.predict(X) and ensemble with other models if you prefer.

Theoretically there is nothing wrong to think of the whole estimator as one model, and use as input in another ensemble (though in your case it looks weird to have multiple voting estimators). In fact, XGB and RF are ensembles themselves.

  • $\begingroup$ The thing is fitting the whole ensemble takes a while. I was unable to access each model's prediction using a Voting Classifier. I solved it by calculating the predictions' mean "manually", using predict_proba in every model and saving it in a seperate array. This way i can add another model and recalculate the voting result by getting the new mean. There should be an easier way though, maybe i'm missing something or this option is just unavaible in Voting ensemble. $\endgroup$ Commented Aug 28, 2023 at 19:10
  • 1
    $\begingroup$ You can easily access the prediction of each model in the voting ensemble via <cls>.named_estimators_.<name of estimator>.predict(X). Check out scikit-learn's example. $\endgroup$
    – lpounng
    Commented Aug 29, 2023 at 1:32
  • $\begingroup$ That's exactly what i needed, i was not able to find it, thank you very much! Unfortunately i can't upvote it yet, but i marked it as resolved $\endgroup$ Commented Aug 29, 2023 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.