2
$\begingroup$

I trained a CNN model and a combined CNN-SVM model for classification. I wanted to compare their performance using ROC curve but I was confused which model is better. How to interpret the given ROC curves ? CNN CNN-SVM

$\endgroup$
  • 2
    $\begingroup$ Your second curve looks unusually regular, it's as if there wasn't enough points to calculate it with precision. $\endgroup$ – Erwan Nov 24 '19 at 23:59
  • $\begingroup$ You mean by enough points the test data ?? Actually in the test data the the number of samples for each class are not equal. $\endgroup$ – root Nov 25 '19 at 19:16
  • $\begingroup$ it's not about the points in the data strictly speaking, but it could be related. the ROC curve is made of the points which correspond to the different possible thresholds in the trade-off between precision and recall. for some reason there seems to be very few of these points in the 2nd case (maybe even only one actually). maybe the method doesn't output a numeric value to be used as threshold? $\endgroup$ – Erwan Nov 25 '19 at 20:21
3
$\begingroup$

If you hear from the area under the curve (AUC), you can find that the first classifier is better as AUC of the first curve is more than the second classifier. To know more about AUC you can find this post useful.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Some other answers alluded to a simplistic interpretation of the ROC curve: The higher the area under the curve, the better the model is at separating positive and negative groups.

The terminology can apply to any two labels, but positive and negative are most commonly used.

So what does the ROC curve plot? From the ROC curve you can measure the ability of the model to tell the two groups apart. Suppose the model produces a prediction $\hat{y}_i \in \mathbb{R}$ for some data. Based on this prediction you should make a decision to label that data as positive or negative. The ROC curve shows the false- and true positive rates of the model, depending on where that threshold is put.

Take for example a perfect model, which can fully separate the two group. There will be a band of possible thresholds $t$ where all of the $\hat{y}_{i \in -} < t$ and $\hat{y}_{i\in+} > t$. In this case, there will be no false positives, while having all true positives. The ROC curve will go through the point $(0,1)$.

If the model is not perfect, when there is overlap between the predictions of the two groups, there will be no threshold $t$ where the true positive rate is 1, and the false positive rate is 0. This means that the curve doesn't go through the left-top corner, but instead forms a curve. The area under this curve shows how good the model is, but that's not all. We can also use the shape of the curve to read what the model struggles with.

There is after all more than one way the data might overlap. If for example, the negative group has a very long tail that overlaps with the positives, that's different from two identical distributions.

Suppose the negative group has a long positive tail so that only the tail overlaps with the positive group. There still is a large part of the negative group that we can correctly label. Similarly, we can catch 100% of the positive group, at the cost of an elevated false positive rate due to that tail. This will be reflected in the ROC curve's shape. Because of this assymetric distribution of classification errors, the ROC curve will be assymetrical as well.

This is a scenario that looks similar to your second curve (with the two labels swapped). If this curve is correct, instead of a possible misspecification like @Erwan commented, You might be dealing with a relatively widely spreaded group that partially overlaps with the other.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The ROC curve shows you the number of items that are correctly classified as being in the positive class versus the number of items that are labeled as positive but are actually negative.

In general, you want to choose the model that has a higher True Positive Rate at a lower False Positive Rate or the ROC curve that converges to 1 the quickest, which means you'd choose the first model.

AUC is a good way to evaluate many ROC curves if they all have a similar shape, but when the AUCs are the same (which I've never come across in practice but could occur) then the shape of the ROC curve becomes important.

| improve this answer | |
$\endgroup$
-1
$\begingroup$

One additional view point i would like to add is that AUC is a ranking metric (what matters is the score order but not the score value itself). so if you order your scores before evaluating it with roc auc metric you can boost your score. Read more about here https://towardsdatascience.com/an-interesting-and-intuitive-view-of-auc-5f6498d87328

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't understand what you mean by "so if you order your scores before evaluating it with roc auc metric you can boost your score." $\endgroup$ – Ben Reiniger Dec 20 '19 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.