Poor Precision-Recall curve for binary classifier trained on balanced data, with imbalanced test data

Question

I have an very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).

I am performing classification using RandomForest. My test set comprises of unbalanced data (since this is how it is expected to be in the real world), and my ROC curve is pretty good, with AUC=0.873. However, my P-R curve suffers, with AUC=0.547.

If I balance the test data, I see a better AUC; however, this is not how the ground truth has behaved in the past, so quoting performance on balanced test data doesn't seem too rational.

What am I doing wrong? Should I train on imbalanced data? Or are there any ways where I can improve the P-R curve?

Answer 1

The precision-recall curve and the ROC curve respond very differently to class imbalance. The precision-recall curve is very sensitive to imbalance while the ROC curve can mask the effect of imbalance.

For the TPR (=recall) and FPR you divide by the true number of samples in each class, while for precision you divide by the number of positive predictions.

If you have few positive samples the ratio between false and true positives becomes more sensitive (compared to false positives [few] and all true negatives [many]).

Depending on your use case, this might be a problem (or not) and should be part of your evaluation.