ROC curve interpretation

Question

I trained a CNN model and a combined CNN-SVM model for classification. I wanted to compare their performance using ROC curve but I was confused which model is better. How to interpret the given ROC curves ?

Answer 1

If you hear from the area under the curve (AUC), you can find that the first classifier is better as AUC of the first curve is more than the second classifier. To know more about AUC you can find this post useful.

Answer 2

Some other answers alluded to a simplistic interpretation of the ROC curve: The higher the area under the curve, the better the model is at separating positive and negative groups.

The terminology can apply to any two labels, but positive and negative are most commonly used.

So what does the ROC curve plot? From the ROC curve you can measure the ability of the model to tell the two groups apart. Suppose the model produces a prediction $\hat{y}_i \in \mathbb{R}$ for some data. Based on this prediction you should make a decision to label that data as positive or negative. The ROC curve shows the false- and true positive rates of the model, depending on where that threshold is put.

Take for example a perfect model, which can fully separate the two group. There will be a band of possible thresholds $t$ where all of the $\hat{y}_{i \in -} < t$ and $\hat{y}_{i\in+} > t$. In this case, there will be no false positives, while having all true positives. The ROC curve will go through the point $(0,1)$.

If the model is not perfect, when there is overlap between the predictions of the two groups, there will be no threshold $t$ where the true positive rate is 1, and the false positive rate is 0. This means that the curve doesn't go through the left-top corner, but instead forms a curve. The area under this curve shows how good the model is, but that's not all. We can also use the shape of the curve to read what the model struggles with.

There is after all more than one way the data might overlap. If for example, the negative group has a very long tail that overlaps with the positives, that's different from two identical distributions.

Suppose the negative group has a long positive tail so that only the tail overlaps with the positive group. There still is a large part of the negative group that we can correctly label. Similarly, we can catch 100% of the positive group, at the cost of an elevated false positive rate due to that tail. This will be reflected in the ROC curve's shape. Because of this assymetric distribution of classification errors, the ROC curve will be assymetrical as well.

This is a scenario that looks similar to your second curve (with the two labels swapped). If this curve is correct, instead of a possible misspecification like @Erwan commented, You might be dealing with a relatively widely spreaded group that partially overlaps with the other.

Answer 3

The ROC curve shows you the number of items that are correctly classified as being in the positive class versus the number of items that are labeled as positive but are actually negative.

In general, you want to choose the model that has a higher True Positive Rate at a lower False Positive Rate or the ROC curve that converges to 1 the quickest, which means you'd choose the first model.

AUC is a good way to evaluate many ROC curves if they all have a similar shape, but when the AUCs are the same (which I've never come across in practice but could occur) then the shape of the ROC curve becomes important.

Answer 4

One additional view point i would like to add is that AUC is a ranking metric (what matters is the score order but not the score value itself). so if you order your scores before evaluating it with roc auc metric you can boost your score. Read more about here https://towardsdatascience.com/an-interesting-and-intuitive-view-of-auc-5f6498d87328