I have an very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).
I am performing classification using RandomForest. My test set comprises of unbalanced data (since this is how it is expected to be in the real world), and my ROC curve is pretty good, with AUC=0.873. However, my P-R curve suffers, with AUC=0.547.
If I balance the test data, I see a better AUC; however, this is not how the ground truth has behaved in the past, so quoting performance on balanced test data doesn't seem too rational.
What am I doing wrong? Should I train on imbalanced data? Or are there any ways where I can improve the P-R curve?