1
$\begingroup$

I am dealing with class imbalance problem. In this case, I am down sampling the majority class lables in the training set.

Among training, validation and test splits, the majority class in training split is down-sampled, and test split is retained as it is. However, should the validation split be downsampled according to the training-set or should it be retained as it is?

This is because the validation set controls the training process.

$\endgroup$
1
$\begingroup$

I would recommend not to downsample the validation set. In the end you care about performance on the test set with the skewed class distribution. Therefore your validation set (used for hyperparameter selection, early stopping etc.) should have the same distribution in my opinion.

Have you considered upsampling the minority class? By downsampling you loose training data, which might contain valuable information and therefore might harm the learning process.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your suggestion. Upsampling by replicating the minority class has not give any performance gain. Currently, I wanted to measure the difference in performance before and after downsampling. $\endgroup$ – Ashwin Geet D'Sa May 18 at 9:49
  • $\begingroup$ Good approach! You could also try to weight the loss function. $\endgroup$ – Tinu May 18 at 9:51
  • $\begingroup$ Yes, will try that. thank you. $\endgroup$ – Ashwin Geet D'Sa May 18 at 9:54
  • $\begingroup$ Glad I could help :) Feel free to accept my answer and help me gain some imaginary internet points :) $\endgroup$ – Tinu May 18 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.