3
$\begingroup$

I plan to use many methods to solve the imbalanced dataset problem on the training set. But I couldn't find any paper that describes how they dealt with the test dataset? I assume that they just tested on the original dataset without any adjustments? Will I need to adjust the threshold on the test set with original imbalanced ratio?

update. Thanks for everyone's response!

I've found a paper that discusses how we should adjust the posterior probability and threshold. Calibrating Probability with Undersampling for Unbalanced Classification

However, I also believe that in practice we can just fit test set directly.

Improve this question
$\endgroup$
1
  • $\begingroup$ I think you should use the test set without any adjustments, because your trained model is going to be applied to imbalanced data. A reason for solving imbalanced data is just to avoid that the prediction of the trained model is trivial. $\endgroup$
    – H. Shindoh
    Mar 26 '17 at 14:34
7
$\begingroup$

You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy. In overall accuracy you find ( number of samples predicted correctly/ total number of samples) in average accuracy, you find the overall accuracy per class and then you find the average of these overall accuracies. When you know that you are working with imbalanced database, where all classes are important, you should use the average accuracy

To understand what this means: imagine you have two classes, class A and class B , and the ratio is 90 to 10 . If you are sampling randomly for the training and testing, then the ratio is still 90:10 in the testing set. If your model is very biased , that predicts all the samples to be class A , then: Overall accuracy = 90% Average accuracy = 50 % ( 100% for class A + 0% for class B) / 2

The overall accuracy is really high but it does not reflect the actual quality of the model. The average accuracy gives you a better indication of the quality

Improve this answer
$\endgroup$
1
$\begingroup$

The idea of balancing the training set + validating the balancing method is for being able to generalize your model that is would discriminate (in classification assignment) better a sample from the minority class, in an unseen and imbalanced test set. Meaning, that the model already adjusted the classification boundaries for considering the minority class underrepresentation and you shouldn't adjust the test set for it.

Finally, you should make sure that both sets are chosen randomly, for having approximately the same representation ratios of examples from each feature

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.