Undersampling for credit card fraud detection before or after Train/Test Split

Question

I have a credit card dataset with 98% transactions are Non-Fraud and 2% are fraud.

I have been trying to undersample the majotrity class before train and test split and get very good recall and precision on the test set.

When I do the undersampling only on training set and test on the independent set I get a very poor precision but the same recall!

My question is :

1. Should I undersample before splitting into train and test , will this mess with the distribution of the dataset and not be representative of the real world?

2. Or does the above logic only apply when oversampling?

Answer 1

Answering your questions, it is important to remember that this kind of transformations, in this case the change in the minority class distribution by oversampling or in the majority class by undersampling, must be only done in the training dataset, so you do not alter the real situation when you want to apply your model (which, at training time, is your test and hold-out set).

About dealing with class imbalance, in addition to resampling methods, you can apply balancing weights to the learning algorithm itself, via parameters like class_weight for some classifiers with scikit-learn (for instance in this one) or via scale_pos_weight with XGBoost (read this)

Answer 2

Do the undersampling after the train/test split and only in the train split: you want to somehow "weigth" your learning algorithm in order to prevent it to be biased towards the majority class, indeed, other techniques can be applied, and always on the train split, what eventually is the data for the learning algorithm. But... your test set should represent the real data distribution. In this case, you will asses if whatever technique you have applied to prevent a biased algorithm towards the majority class has not eventually affected its performance on it.

Some other approaches for this problem (fraud vs non-fraud):

• Anomally detection
• One-class classifies: One Class SVM