How to tell if downsampling helped model performance

Question

Fitting a logistic classifier to imbalanced data. My target variable is 5% 1, 95% 0. As such, I think it's probably best to use the PR-AUC to evaluate the model rather than ROC-AUC. I get a PR-AUC of 0.1, which is better than nothing I guess.

Another way to potentially increase performance is to downsample the majority class (or upsample the minority or some combination but let's stick with downsampling).

The problem is that how do I tell if the downsampling actually helped model performance? unlike ROC-AUC which falls between 0.5 for a random model and 1 for a great model, PR-AUC is relative to the percentage of data in the positive class. Because the percentage of the positive class is different in these models by design, how can I compare them?

Answer 1

You never downsample your test data. Test data should maintain same %age as original distribution of classes. You compare test reseults before and after sampling to see if it works

Answer 2

This sounds like the AUPRC is being done on the training (downsampled) set. Compare the models on the same validation set.

AUPRC does not solve all issues either. It is just another metric with strengths and weaknesses. Interpretation, getting (un)lucky with the highest score observation, and some issues pointed out here. Just the highest scored observation may change the AUPRC and cause you a different interpretation.

Also, class imbalance may not ned to be worried about. There are a lot of posts on this site and stats.stackexchange.com that show do not worry about class imbalance. Some links I gave are also in my comment here.

If your model is deployed for a long time between trainings, the downsampling may make the scoring less stable. Information is being thrown away that may be useful for stability. I have found this in my work.