0
$\begingroup$

Fitting a logistic classifier to imbalanced data. My target variable is 5% 1, 95% 0. As such, I think it's probably best to use the PR-AUC to evaluate the model rather than ROC-AUC. I get a PR-AUC of 0.1, which is better than nothing I guess.

Another way to potentially increase performance is to downsample the majority class (or upsample the minority or some combination but let's stick with downsampling).

The problem is that how do I tell if the downsampling actually helped model performance? unlike ROC-AUC which falls between 0.5 for a random model and 1 for a great model, PR-AUC is relative to the percentage of data in the positive class. Because the percentage of the positive class is different in these models by design, how can I compare them?

$\endgroup$

2 Answers 2

1
$\begingroup$

You never downsample your test data. Test data should maintain same %age as original distribution of classes. You compare test reseults before and after sampling to see if it works

$\endgroup$
1
$\begingroup$

This sounds like the AUPRC is being done on the training (downsampled) set. Compare the models on the same validation set.

AUPRC does not solve all issues either. It is just another metric with strengths and weaknesses. Interpretation, getting (un)lucky with the highest score observation, and some issues pointed out here. Just the highest scored observation may change the AUPRC and cause you a different interpretation.

Also, class imbalance may not ned to be worried about. There are a lot of posts on this site and stats.stackexchange.com that show do not worry about class imbalance. Some links I gave are also in my comment here.

If your model is deployed for a long time between trainings, the downsampling may make the scoring less stable. Information is being thrown away that may be useful for stability. I have found this in my work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.