0
$\begingroup$

I am working on a binary classification problem which dataset has about 5% of positive class samples. I split the dataset, 70% for training and 30% for testing. I used the test data only once for reporting the performance of the model.

Due to this imbalance, I used SMOTE to upsampling the minority class in the training dataset. In addition I used CV and grid search to optimize model performance following the suggestions from how to upsample, CV, and gridsearch to avoid data leakage.

Assuming that I am handling the training procedure correctly, I wonder how to report the classification results in the test data. My understanding is that for imbalanced datasets you should used AUPRC (see nice explanation here). So, if I address the imbalanced problem in training, do I need to report results using AUPRC or it is ok to used traditional ROC-AUC?

Thanks for your help in advance.

$\endgroup$
1

1 Answer 1

0
$\begingroup$

Your training and test data are supposed to come from the same distribution, therefore if the training dataset is unbalanced so the test one (probably) will.

The fact you oversampled the minority class in the training dataset does not affect the unbalance in the test dataset. So yes, you should use specific metrics to account for the unbalancedness also when reporting results on the test set as traditional ones may not reflect a lot the differences in performances between various predictors.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the quick reply, @DaSim. Yes, the test dataset is imbalanced too. I think using AUPRC makes sense given that SMOTE is only used in training. Thanks again. $\endgroup$
    – Paul
    Aug 17, 2022 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.