Fitting a logistic classifier to imbalanced data. My target variable is 5% 1, 95% 0. As such, I think it's probably best to use the PR-AUC to evaluate the model rather than ROC-AUC. I get a PR-AUC of 0.1, which is better than nothing I guess.
Another way to potentially increase performance is to downsample the majority class (or upsample the minority or some combination but let's stick with downsampling).
The problem is that how do I tell if the downsampling actually helped model performance? unlike ROC-AUC which falls between 0.5 for a random model and 1 for a great model, PR-AUC is relative to the percentage of data in the positive class. Because the percentage of the positive class is different in these models by design, how can I compare them?