Individual models gives quite same distribution on Test set, whereas Ensembling gives better result but very different distribution

Question

I am working on a binary classification problem with unbalanced data (17% for positive class).

The problem is as following: My three individual models when predicting on the test set (for which I don't have the labels) gives quite similar distribution as for Train set.

But ensemling these models, while giving slighltly better result (F1-score), it drastically changes the distribution on Test set going from ~20% to 5%.

My question is :

I am confused between choosing the best individual model which maintains almost same distribution but lose some efficiency Or The ensembled one who gives really different distribution

And I have no Idea about the Test set distribution.

Thanks for any help

Answer 1

Depending on the size of the test and training set, and how they were sampled, their distributions may not be the same. if the sets aren't very big or weren't randomly sampled then their distributions won't necessarily be the same, and may not correspond to the distribution of the population

One way to test this is to compare the distributions of the other variables in the test and training sets.

In terms of selecting individual models or the ensemble for prediction, I would recommend trying the individual models and the ensembled combination in k-fold validation, and selecting whichever approach results in the best performance. This way you are using the entire training set for training and validation, and the assessment of this performance should be the best approximation of test set performance.