1
$\begingroup$

I have an imbalanced dataset and I'm using XGBoost to do binary classification. I used down sampling together with target and one hot encoding for train data. For test data I once used just the encodings and left it unbalanced and once tried with a balanced test dataset.

The ROC AUC score was quite higher for the imbalanced test data than the balanced one. How is this possible? I thought for the ROC AUC score there should not be any difference?

Improve this question
$\endgroup$
3
  • $\begingroup$ I still need to add something: this is only the case if I use cross validation. So for CV the ROC AUC score is lower if I balance the test data. If I balance the test data and just fit and predict and then calculate the score (without CV) then the ROC AUC score is higher and the same as if I leave the test data imbalanced (with or without CV). It seems like it also depends on how many folds i choose: if I choose less folds the score is also higher. Why does CV behave like this with balanced test data? $\endgroup$
    – corianne1234
    Aug 16, 2019 at 13:52
  • $\begingroup$ Are you cross-validating for model selection? When cross-validating, are you reporting the AUC on the separate test set, or the (average over the) test folds? What are the AUC values? With more/fewer folds, are the scores across folds consistent? $\endgroup$
    – Ben Reiniger ♦
    Aug 16, 2019 at 14:16
  • $\begingroup$ Yes, I use CV for model selection. I'm using just one set for cross validation and then take the mean of the test scores. The AUC values are not really consistent with more/fewer folds. I do downsampling before CV and not during CV, so the test sample is also balanced (just like train) but I thought it wouldn't matter if the test set is balanced or not? $\endgroup$
    – corianne1234
    Aug 16, 2019 at 14:47

1 Answer 1

Reset to default
2
$\begingroup$

ROC normally reports higher scores for imbalanced data sets because it does not take into account false discoveries. For this reason, it is recommended to only use ROC when the positive and negative sets are roughly equal in size. Otherwise, the suggestion is to compute the area under the Precision-Recall Curve instead.

You can read a bit about the reasons here.

https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/

However, ROC curves can present an overly optimistic view of an algorithm’s performance if there is a large skew in the class distribution. […] Precision-Recall (PR) curves, often used in Information Retrieval , have been cited as an alternative to ROC curves for tasks with a large skew in the class distribution.

— The Relationship Between Precision-Recall and ROC Curves, 2006.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.