1
$\begingroup$

I tried out a few models on a highly imbalanced sample (~2:100) where I can get decent AUC from ROC (test sample). But when I plot precision-recall (test sample), it looks horrible. Kind of like the worse PR curve in box (d).

This article contains the picture from below and describes that ROC is better suited since it is invariant to class distribution.

My question is if there's anything than can be done to improve precision-recall?

ROC vs Precision-Recall Curve

$\endgroup$
0
$\begingroup$

I am not sure if you are asking how to improve the power of the models or how to better represent the effectiveness of the models on the minority class. These are information retrieval measures that are sometimes adequate in other circumstances. If you are working on an information retrieval process, this is probably fine. If it is more of a statistical analysis you are working on, perhaps sensitivity (which is the same as recall) and specificity would be better metrics to describe your results. Maybe your model is useful in that it can rule out the negative. These metrics can also be evaluated relative to the prior probability of the minor class in the data.

Sensitivity and specificity have an additional benefit if you are trying to communicate your results with an audience that reviews a lot of medical test results as these measures are used most frequently in that field.

HTH

$\endgroup$
  • $\begingroup$ thanks for taking the time to answer. As you mentioned, Sensitivity and Specificity are definitely valid and are accounted in ROC where FPR = 1 - Specificity. I'm actually looking at credit data and trying to classify which applicant will go into default/stay current. So am concerned about how many profitable customers will be turned away as well. As you mentioned, I'd like to represent the effectiveness of models on minority class while also keeping in mind the cost of turning away good customers $\endgroup$ – rayven1lk Oct 25 '18 at 15:40
0
$\begingroup$

Probably yes. If you use Logistic Regression with L2, change C from 1 (default in SKLearn) to the Power of 10. E.g. C from 1 to 10, 100, 1000 ....

C is the inverse of regularization strength. By increasing C you decrease bias and increase variance. In other words, you're probably going to overfit.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.