Trying to answer
When dealing with class imbalance ROC is not a good criteria. There is a post on Kaggle about Credit Card Fraud Detection with experiment over the class imbalance, possible ways of mitigating the imbalance, better metrics and code in python for every experiment.
Since it is a really long post (actually it is a notebook on class imbalance and roc), but here I quote author's conclusion while Comparing PR curves to ROC:
For a PR curve, a good classifer aims for the upper right corner of
the chart but upper left for the ROC curve.
While PR and ROC curves use the same data, i.e. the real class labels
and predicted probability for the class lables, you can see that the
two charts tell very different stories, with some weights seem to
perform better in ROC than in the PR curve.
While the blue, w=1, line performed poorly in both charts, the black,
w=10000, line performed "well" in the ROC but poorly in the PR curve.
This is due to the high class imbalance in our data. ROC curve is not
a good visual illustration for highly imbalanced data, because the
False Positive Rate ( False Positives / Total Real Negatives ) does
not drop drastically when the Total Real Negatives is huge.
Whereas Precision ( True Positives / (True Positives + False
Positives) ) is highly sensitive to False Positives and is not
impacted by a large total real negative denominator.
The biggest difference among the models are at around 0.8 recall rate.
Seems like a lower weight, i.e. 5 and 10, out performs other weights
significantly at 0.8 recall. This means that with those specific
weights, our model can detect frauds fairly well (catching 80% of
fraud) while not annoying a bunch of customers with false positives
with an equally high precision of 80%.
Without further tuning our model, and of course we should do cross
validation for any real model tuning/validation, it seems like a
vanilla Logistic Regression is stuck at around 0.8 Precision and
Recall.
So how do we know if we should sacrifice our precision for more
recall, i.e. catching fraud? That is where data science meets your
core business parameters. If the cost of missing a fraud highly
outweighs the cost of canceling a bunch of legit customer
transactions, i.e. false positives, then perhaps we can choose a
weight that gives us a higher recall rate. Or maybe catching 80% of
fraud is good enough for your business if you can minimize also
minimize the "user friction" or credit card disruptions by keeping our
precision high.
My conclusion:
Even though your PR curve looks bad, given the class imbalance, that is the best metric and you should try to improve so your model is best fitted for your application
Observation:
You have a dataset with 100 samples and only 2 are from $C_1$, gathered right so far?
Supose now that from these 2, one is a complete outlier to its one class $C_1$, he provides more harm than good in your model fitting, and since he accounts for 50% of your $C_1$ samples he is going to make your model perform porly.
