In a binary classification problem,

I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem.

I tried many pipelines, automls, and threshold tuning etc and I am not getting "good" recall and precision at the same time. (I would like to get at least .75 recall and precision at the same time if possible.)

I assume class overlapping is there (no checking done yet) and I applied instance hardness threshold for removing hard to predict instances from class 0 (majority class) as undersampling. Now my class distribution is like this: 0:1530, 1:1500

Then I split the undersampled data into train and test. I know it is a bad idea to undersample data before train test split, since the test set will not be the true representation of real-world problem as stated in this stackexchange question.

But I wonder, is it right to do something like instance hardness threshold undersampling before training? After doing so, I am getting all scores above 95%. But it is too optimistic!! Kindly help me to clarify the doubt regarding this!

  • $\begingroup$ if you dont care about the hard instances, then all is fine. Else you should try to include them in test set $\endgroup$
    – Nikos M.
    Jun 13 at 3:57
  • $\begingroup$ But if I include it in test set, since such kind of instances not seen while training, I am not getting good performance. $\endgroup$
    – DOT
    Jun 13 at 6:00
  • $\begingroup$ it is all about whether you care about the hard instances. If you dont care so much, then leave them out. Else include them both in train and test (at least a portion of them) $\endgroup$
    – Nikos M.
    Jun 13 at 6:02
  • $\begingroup$ Thank you very much! $\endgroup$
    – DOT
    Jun 13 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.