3
$\begingroup$

I am dealing with class imbalance problem. In this case, I am down sampling the majority class lables in the training set.

Among training, validation and test splits, the majority class in training split is down-sampled, and test split is retained as it is. However, should the validation split be downsampled according to the training-set or should it be retained as it is?

This is because the validation set controls the training process.

$\endgroup$

1 Answer 1

2
$\begingroup$

I would recommend not to downsample the validation set. In the end you care about performance on the test set with the skewed class distribution. Therefore your validation set (used for hyperparameter selection, early stopping etc.) should have the same distribution in my opinion.

Have you considered upsampling the minority class? By downsampling you loose training data, which might contain valuable information and therefore might harm the learning process.

$\endgroup$
4
  • $\begingroup$ Thanks for your suggestion. Upsampling by replicating the minority class has not give any performance gain. Currently, I wanted to measure the difference in performance before and after downsampling. $\endgroup$ Commented May 18, 2020 at 9:49
  • $\begingroup$ Good approach! You could also try to weight the loss function. $\endgroup$
    – Tinu
    Commented May 18, 2020 at 9:51
  • $\begingroup$ Yes, will try that. thank you. $\endgroup$ Commented May 18, 2020 at 9:54
  • $\begingroup$ Glad I could help :) Feel free to accept my answer and help me gain some imaginary internet points :) $\endgroup$
    – Tinu
    Commented May 18, 2020 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.