In a binary classification problem,
I have a slightly unbalanced medical dataset with class distribution: 0:5600, 1:1500 0 without a problem and 1 with a problem.
I tried many pipelines, automls, and threshold tuning etc and I am not getting "good" recall and precision at the same time. (I would like to get at least .75 recall and precision at the same time if possible.)
I assume class overlapping is there (no checking done yet) and I applied instance hardness threshold for removing hard to predict instances from class 0 (majority class) as undersampling. Now my class distribution is like this: 0:1530, 1:1500
Then I split the undersampled data into train and test. I know it is a bad idea to undersample data before train test split, since the test set will not be the true representation of real-world problem as stated in this stackexchange question.
But I wonder, is it right to do something like instance hardness threshold undersampling before training? After doing so, I am getting all scores above 95%. But it is too optimistic!! Kindly help me to clarify the doubt regarding this!