Correctly evaluate model with oversampling and cross-validation

Question

I'm dealing with a classic case of dataset with binary imbalanced target (event 3%, non event 97%). My idea is to apply some sort of sampling (over/under, SMOTE etc.) to address the issue.

As I see, the correct way of doing this is to sample ONLY the train set, in order to have a test performance that is more similar to reality. Moreover, I want to use CV for hyperparameters tuning. So, the tasks in order are

1. Divide dataset into train-test
2. Perform the 5 fold-CV, as...
3. Sample the "training" portion of the CV
4. Sample the "validating" portion of the CV
5. Train the model on the "training"
6. Validate it on the "validating"
7. Repeat 3-6 5 times
8. Evaluate performances on test

My doubt is: how can I compare the CV performances with the test, since the former are based on sampled data and the latter does not?

An idea is to skip 4 and sample only "training" portion, but in this case how can I compare the "training" with the "validating"?

EDIT: added target ratio + typo.

Answer 1

The first issue is why do you need to use SMOTE? Imbalanced datasets, provided they are sufficiently large, do not present a significant problem for statistical classifiers or machine learning methods. If you have an imbalanced dataset, quite often the optimal accuracy is obtained by assigning everything to the majority class. If that is not acceptable, it is an indication that the minority class is more "important" in some sense than the majority class (i.e. there ought to be a higher cost for missclassifying a minority class pattern as belonging to the majority class than vice versa). In other words, accuracy is not the right performance metric and you need to look at the expected loss (effectively a weighted accuracy - weighted according to the misclassification costs). So rather than using SMOTE, it would be better to see if you can work out what the misclassification costs actually should be and incorporate those into the classifier (either by changing the threshold probability for a probabilistic classifier, or by weighting the positive and negative patterns unequally in the training criterion). Most often "class imbalance problems" are just "cost sensitive learning problems" in disguise.

Note that SMOTE was originally developed in the context of very primitive classifier systems, such as single decision trees or RIPPER, that were prone to over-fitting if minority examples were simply resampled. The generation of synthetic examples acts to "blur" the minority examples, so they are more difficult to overfit. Modern classifier systems have effective means of avoiding overfitting, such as regularisation, so it is questionable whether the rather odd way in which SMOTE generates synthetic examples is a good idea for modern methods.

If you are tuning hyper-parameters to optimise operational peformance, then the "test" folds in cross-validation should be representative of operational conditions, so you should not be applying SMOTE to them or resampling, if your original dataset was representative of operational conditions.

Answer 2

I believe, the sequence for combination of CV and SMOTE should be as below.

1. Perform the 5 fold-CV ( Loop through for each fold )
2. Training Sample and Testing Sample ( for each fold )
3. Smote Training Samples
4. Train the model on the "training"
5. Prediction ( test samples )
6. Evaluate performances on test
Repeat for next fold