1
$\begingroup$

I have a dataset where around 20% of the data is the positive class and 80% of the data is the negative class. When I undersample and train my classifier on a balanced dataset and test on a balanced dataset, the results are pretty ok. However, if I train on the balanced dataset and test on an imbalanced dataset that replicates the real world (80-20 split) the metrics are not great. Should I train the model on the original imbalanced dataset if I want it to perform well on real world test data that is also imbalanced.

$\endgroup$
  • $\begingroup$ There may be some good information and discussion here in this thread datascience.stackexchange.com/questions/8895/… although I do feel that Erwan has a perfectly acceptable answer as well. $\endgroup$ – Dylan Jan 21 at 19:14
  • $\begingroup$ I'm voting to close this question because this question doesn't show a lot of research effort and already has many related answers across the network. If you have a less general question you may edit the question and vote to reopen. $\endgroup$ – oW_ Jan 21 at 22:09
1
$\begingroup$

When I undersample and train my classifier on a balanced dataset and test on a balanced dataset, the results are pretty ok

It's not surprising that the results are good since the job is easier in this case. It's actually a mistake to test on the artificially balanced dataset, since it's not a fair evaluation of how the system will perform with real data.

Should I train the model on the original imbalanced dataset if I want it to perform well on real world test data that is also imbalanced.

Both training on the original dataset or the balanced dataset are valid methods, choosing between the two options is a matter of design and performance. It's often a good idea to try both and then pick the one which performs better than the other on the real imbalanced dataset.

|improve this answer|||||
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.