I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.
I have been trying to undersample the majotrity class before train and test split and get very good recall and precision on the test set.
When I do the undersampling only on training set and test on the independent set I get a very poor precision but the same recall!
My question is :
Should I undersample before splitting into train and test , will this mess with the distribution of the dataset and not be representative of the real world?
Or does the above logic only apply when oversampling?