# ROC AUC score is better if test data is imbalanced

I have an imbalanced dataset and I'm using XGBoost to do binary classification. I used down sampling together with target and one hot encoding for train data. For test data I once used just the encodings and left it unbalanced and once tried with a balanced test dataset.

The ROC AUC score was quite higher for the imbalanced test data than the balanced one. How is this possible? I thought for the ROC AUC score there should not be any difference?

• I still need to add something: this is only the case if I use cross validation. So for CV the ROC AUC score is lower if I balance the test data. If I balance the test data and just fit and predict and then calculate the score (without CV) then the ROC AUC score is higher and the same as if I leave the test data imbalanced (with or without CV). It seems like it also depends on how many folds i choose: if I choose less folds the score is also higher. Why does CV behave like this with balanced test data? – corianne1234 Aug 16 '19 at 13:52
• Are you cross-validating for model selection? When cross-validating, are you reporting the AUC on the separate test set, or the (average over the) test folds? What are the AUC values? With more/fewer folds, are the scores across folds consistent? – Ben Reiniger Aug 16 '19 at 14:16
• Yes, I use CV for model selection. I'm using just one set for cross validation and then take the mean of the test scores. The AUC values are not really consistent with more/fewer folds. I do downsampling before CV and not during CV, so the test sample is also balanced (just like train) but I thought it wouldn't matter if the test set is balanced or not? – corianne1234 Aug 16 '19 at 14:47