I am working on a binary classification problem with unbalanced data (17% for positive class).

The problem is as following: My three individual models when predicting on the test set (for which I don't have the labels) gives quite similar distribution as for Train set.

But ensemling these models, while giving slighltly better result (F1-score), it drastically changes the distribution on Test set going from ~20% to 5%.

My question is :

I am confused between choosing the best individual model which maintains almost same distribution but lose some efficiency Or The ensembled one who gives really different distribution

And I have no Idea about the Test set distribution.

Thanks for any help


Depending on the size of the test and training set, and how they were sampled, their distributions may not be the same. if the sets aren't very big or weren't randomly sampled then their distributions won't necessarily be the same, and may not correspond to the distribution of the population

One way to test this is to compare the distributions of the other variables in the test and training sets.

In terms of selecting individual models or the ensemble for prediction, I would recommend trying the individual models and the ensembled combination in k-fold validation, and selecting whichever approach results in the best performance. This way you are using the entire training set for training and validation, and the assessment of this performance should be the best approximation of test set performance.

  • $\begingroup$ Thanks Fab, 1. Indeed datasets are of small size 2. I have one variable to compare, and the distribution is not the same between Train and Test 3. Nice Idea I will try CrossValidate and inform you $\endgroup$ Jan 31 '21 at 12:30
  • $\begingroup$ Thanks @Fab your help was very valuable $\endgroup$ Feb 2 '21 at 21:49
  • $\begingroup$ you're welcome :) $\endgroup$
    – Fab
    Feb 5 '21 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.