Estimating test AUC using k-fold CV for imbalanced classification problem

Question

I have an imbalanced classification problem.

I first partitioned my data into a training set (Dataset A) and a test set (Dataset B).

I then used the R package ROSE to simultaneously undersample the majority class in Dataset A and oversample the minority class. This produced a balanced set (Dataset C) which has as many rows as Dataset A.

I have trained my model on Dataset C and computed training AUC. I now want to estimate (i.e., not using Dataset B) the test AUC using k-fold cross-validation. How should I best do this?

Answer 1

The test AUC is the AUC you find after predicting the held out test set (dataset B).

You have split your dataet into A = training set and B = test set. You then used downsampling (which is questionable, but I won't get into this) on dataset A to get what you refer to as dataset C. You should then train your classifier on dataset C and evaluate its performance on dataset B.

If what you are referring to is a validation set (i.e. not the test set = dataset B) then what you need to do is as follows: split your dataset into A = outer train, and B = test. Then, split A again into C = inner train, D = validation. Downsample dataset C, train your classifier on C only, then validate on D. Once you have optimized to D, then downsample A, retrain your classifier on A (using the exact same model building process as you did with C and D), and report your final, unbiased performance measure on B. Repeat this entire process if AUC scores are volatile to get more estimates of final model performance.

Since you have unbalanced data, I strongly recommend stratified sampling when creating your validation and test sets.

Answer 2

Cross validation is normally done on training dataset as oppose to a method of measuring calculation on test set.

If you want to use a 5-fold cross validation on training dataset, this means your training dataset will be split into 5 parts and training on 4 portion of data while testing on the rest for 5 times. The prediction from the 1 portion (with 5 times) would be used to calculate the chosen measure as cross validated score.

ROSE package looks like does not provide cross validation function, but you can write your own function to implement:

 for( i in 1:5){
    train on 80% data
    predict on 20% data and save prediction
 }