11
$\begingroup$

I am building a binary classification model for imbalanced data (e.g., 90% Pos class vs 10% Neg Class).

I already balanced my training dataset to reflect a a 50/50 class split, while my holdout (training dataset) was kept similar to the original data distribution (i.e., 90% vs 10%). My question is regarding the validation data used during the CV hyperparameter process. During each iteration fold should:

1) Both the training and test folds be balanced

or

2) The training fold should be kept balanced while the validation fold should be made imbalanced to reflect the original data distribution and holdout dataset.

I am currently using the 1st option to tune my model; however, is this approach valid given that the holdout and validation datasets have different distributions?

$\endgroup$
1
  • $\begingroup$ Misconceptions abound when it comes to class imbalance. I suggest reading this Meta post and the links within. $\endgroup$
    – Dave
    Aug 3, 2022 at 23:04

3 Answers 3

8
$\begingroup$

Both test and validation datasets should have the same distribution. In such a case, the performance metrics on the validation dataset are a good approximation of the performance metrics on the test dataset. However, the training dataset can be different. Also, it is fine and sometimes helpful to balance the training dataset. On the other hand, balancing the test dataset could lead to a bias estimation from the performance of the model because the test dataset should reflect the original data imbalance. As I mentioned at the beginning the test and validation datasets should have the same distribution. Since balancing the test dataset is not allowed, the validation dataset can not be balanced too.

Additionally, I should mention that when you balance the test dataset, you will get a better performance in comparison to using an unbalanced dataset for testing. And of course, using a balanced test set does not make sense as explained above. So, the resulted performance is not reliable unless you use an unbalanced dataset with the same distribution of classes as the actual data.

$\endgroup$
4
  • $\begingroup$ how can you use the pipeline in Sklearn to just balance the training folds but not the validation fold. Are you referring to using SMOTE or down-, up-sampling methods? $\endgroup$ Jun 16, 2020 at 1:52
  • $\begingroup$ I mentioned k-fold cross-validation. In that case, we need to have an unbalanced validation fold; while the rest of the folds (k-1 folds) need to be balanced. I remove that line as it might lead to confusion. $\endgroup$
    – nimar
    Jun 16, 2020 at 4:22
  • $\begingroup$ Thank you for the clarification. I also created a follow up question (datascience.stackexchange.com/questions/76107/…) for additional clarification on the recommended training schema, I would appreciate your additional input. $\endgroup$ Jun 16, 2020 at 16:08
  • $\begingroup$ I have a question. To avoid confusion, here's the terminology I use: train, test (used to make sure the model is not overfitting and help choosing parameters) and validation (the sample that simulates the data in production and should reflect the actual performance on the model). In my opinion, the main goal of having the test sample is to check whether the model is overfitting the training data or not. If I'm not allowed to balance it, the performance of training and test samples will be completely different and I'll not be able to tell if the model is generalizing. How to proceed then? $\endgroup$ Dec 7, 2021 at 15:07
2
$\begingroup$

In my opinion the validation set should follow the original imbalanced distribution: the goal is ultimately to apply the model to the real distribution so the hyper-parameters should be chosen to maximize performance for this distribution.

But since I'm not completely sure I'd suggest trying both options, and adopt the one which gives the best performance on the test set.

$\endgroup$
2
  • 1
    $\begingroup$ At one side you are right but another as u said above validation set should follow the imbalance distribution that I disagree because by doing that we can not get degree of generalization of model. What u say about it? $\endgroup$ Jun 16, 2020 at 4:46
  • 1
    $\begingroup$ @GauravKoradiya well I think there are arguments for the two options. in doubt I would try both, as I said. $\endgroup$
    – Erwan
    Jun 16, 2020 at 12:04
0
$\begingroup$

Just sharing what I believe is the reasoning behind the need for a balanced dataset. { Assuming we are talking about a supervised classification :) }

The training dataset is the only piece of data that "teaches" the model how to perform the classification. If you train the model with an unbalanced dataset (A:90; B:10), the model could be lazy enough to classify everything as A and the accuracy will be 90% without the ability to distinguish A and B. The loss function won't be able to guide the training steps towards the real ability to generalize. Therefore, balancing the training dataset forces the model to learn the underlying reasons to classify something as A or B.

But the validation and test should reflect the real distribution of the data. Validation shows when the model is getting too specialized in the training dataset and losing the ability to generalize. Therefore, it is the connection between what the training is producing and the test that will hopefully make good use as a crystal ball once the model is defined.

The test is used after the model selection and this is the first time data don't give you any hint of how the model should be. You can see this part of the data as "the future" from the model perspective.

It's worth mentioning if the problem is focused on outlier detection, it may require more sophisticated approaches.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.