5
$\begingroup$

I have a fairly imbalanced dataset for default-risk credit scoring (2:98). Both costs are fairly important i.e False negative means loss from default and false positive is a lost-revenue opportunity.

I tried training some models on the imbalanced set and tried over/under-sampling approaches as well as weight-balancing based on class. All of these approaches led to a decent ROCAUC. However, the precision-recall curves have looked horrible overall.

But if I were to balance the test set, the precision-recall curve looks much better. Why does this happen? And is there some adjustment I should be applying to improve precision-recall under the imbalanced data distribution?

$\endgroup$
  • 2
    $\begingroup$ Might be that the classifier learns that it should predict the majority class since there are so many more samples for training. Which yields good accuracy but bad precision. $\endgroup$ – Isbister Oct 29 '18 at 15:42
  • $\begingroup$ @Isbister, thanks for the insight. I agree with what you said, but I tried sampling approaches to balance the training set which would help the classifier avoid predicting the majority class $\endgroup$ – rayven1lk Oct 29 '18 at 15:48
  • 1
    $\begingroup$ Can you post some data and your code so that we can see how you are approaching the problem? It seems as though the data is just too imbalanced and you should consider anomaly detection instead. $\endgroup$ – JahKnows Oct 30 '18 at 0:02
1
$\begingroup$

Trying to answer

When dealing with class imbalance ROC is not a good criteria. There is a post on Kaggle about Credit Card Fraud Detection with experiment over the class imbalance, possible ways of mitigating the imbalance, better metrics and code in python for every experiment.

Since it is a really long post (actually it is a notebook on class imbalance and roc), but here I quote author's conclusion while Comparing PR curves to ROC:

ROC and PR Curves

For a PR curve, a good classifer aims for the upper right corner of the chart but upper left for the ROC curve.

While PR and ROC curves use the same data, i.e. the real class labels and predicted probability for the class lables, you can see that the two charts tell very different stories, with some weights seem to perform better in ROC than in the PR curve.

While the blue, w=1, line performed poorly in both charts, the black, w=10000, line performed "well" in the ROC but poorly in the PR curve. This is due to the high class imbalance in our data. ROC curve is not a good visual illustration for highly imbalanced data, because the False Positive Rate ( False Positives / Total Real Negatives ) does not drop drastically when the Total Real Negatives is huge.

Whereas Precision ( True Positives / (True Positives + False Positives) ) is highly sensitive to False Positives and is not impacted by a large total real negative denominator.

The biggest difference among the models are at around 0.8 recall rate. Seems like a lower weight, i.e. 5 and 10, out performs other weights significantly at 0.8 recall. This means that with those specific weights, our model can detect frauds fairly well (catching 80% of fraud) while not annoying a bunch of customers with false positives with an equally high precision of 80%.

Without further tuning our model, and of course we should do cross validation for any real model tuning/validation, it seems like a vanilla Logistic Regression is stuck at around 0.8 Precision and Recall.

So how do we know if we should sacrifice our precision for more recall, i.e. catching fraud? That is where data science meets your core business parameters. If the cost of missing a fraud highly outweighs the cost of canceling a bunch of legit customer transactions, i.e. false positives, then perhaps we can choose a weight that gives us a higher recall rate. Or maybe catching 80% of fraud is good enough for your business if you can minimize also minimize the "user friction" or credit card disruptions by keeping our precision high.

My conclusion:

Even though your PR curve looks bad, given the class imbalance, that is the best metric and you should try to improve so your model is best fitted for your application

Observation:

You have a dataset with 100 samples and only 2 are from $C_1$, gathered right so far?

Supose now that from these 2, one is a complete outlier to its one class $C_1$, he provides more harm than good in your model fitting, and since he accounts for 50% of your $C_1$ samples he is going to make your model perform porly.

| improve this answer | |
$\endgroup$
1
$\begingroup$

This is indeed the expected behavior.

(N.B. You should not balance test sets, since they are supposed to inform you about performance on unseen, original-distribution data.
https://stats.stackexchange.com/a/258974/232706
In a binary classification, should the test dataset be balanced? )

A quick example may be clearer than being general. Let's fix a prediction threshold, and hence a point on the PR curve. Say your test set has 100 positive examples and 1000 negative ones, and your model (with the given threshold) has 80% recall and 40% precision. Then there are 80 TP, 20 FN, 160 FP, and 840 TP. Now you downsample to balanced data*, throwing away a random 900 negative samples (drastic, but to make the point). The model's predictions don't change, just we've lost some samples. We expect to lose 144 of the FP, keeping 16; and lose 756 TN, keeping 84. Now our recall is the same, but precision jumps from 40% to 83.3% ! On the PR graph, we've shifted the point on the curve way up! And of course this happens at each point, independently; so the AU(PR)C will increase substantially.

*Similar things should happen with other resampling balancing methods, provided the resampling happens evenly distributed over your predictions.

| improve this answer | |
$\endgroup$
0
$\begingroup$

One potential reason is that your dataset is too small and your algorithm doesn't have enough samples to 'learn'.

If you 'balance' your dataset, the algorithm has enough data to 'learn' and therefore your precision / recall increases.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.