2
$\begingroup$

So I came across a bioinformatics paper, where I found a line which says:

One potential problem with using a training set with equal numbers of positive and negative examples in cross-validation is that it can artificially inflate performance estimates because the number of false-positive classifications is proportional to the number of examples classified. So applying these methods to all proteins in an organism may result in a large number of false-positive identifications.

I am unable to understand how classification of balanced dataset is a problem. Can someone please explain this to me?

$\endgroup$
2
$\begingroup$

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't work well because of changing the distribution of each class. In cases like a disease prediction where the number of each class varies for different places, a model that is trained in the U.S won't work in African countries at all. The reason is that the distribution of classes has been changed. So in such cases that usually the negative and positive classes are not balanced in practice, balancing them will cause the problem of distribution changes. On these occasions, people usually use the real data-set which is not balanced and use F1 score for evaluation.

$\endgroup$
2
  • $\begingroup$ So then the entire idea of oversampling data would be invalid since oversampling would cause the problem of distribution change in any dataset? $\endgroup$ – girl101 Mar 28 '18 at 4:36
  • $\begingroup$ @girl101 Unfortunately I've not done oversampling in practice. What I said was based on my practice, although is a common issue. $\endgroup$ – Media Mar 28 '18 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.