Significance of comparing Receiver Operating Characteristic (ROC) curves

Question

An ROC curve plots the true positive rate (sensitivity) as function of the false positive rate (100-specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. This would permit one to compare the goodness of the model when varying the given parameter (e.g. number of trees in a Random Forest) to help optimize that particular classifier's parameters. Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)? I came across an instance of such an example on the second last slide here and was trying to understand the significance of these curves. As far as I am aware, the graph displayed appears to not be a necessarily fair comparison when comparing the area under the curve while tuning different parameters in different types of classifiers.

Answer 1

Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)?

Yes, because you can clearly see which model is performing best overall. The more you get close to the top left corner of the graph, the better your model is. (the blue line in your example, Random Forest)

If you compute the Area under the ROC curve in the example, you'll get:

AUC(RF) > AUC(MLP) > AUC(SVM)

So AUC is a good index to show which curve is "higher".

Now, sometimes you could be interested in a model that has a "steeper" ROC curve in the bottom left corner (higher chance of improving TPR, with low cost in term of FPR), and maybe that model isn't the best in term of AUC. (Not the case in your example)

In that case you would not choose AUC as a term of model choice, because it doesn't select the "best" model for you.

Overall the ROC curve between different models helps you a lot, showing not only the performance over different thresholds, but also comparing different solutions simultaneously.

Edit:

Each model gives a number for each observation that is the probability to belong in one class (a number p between 0 and 1). Say you get p = 0.3 for one observation, then with a threshold of 0.5 you will label that observation to one of the two classes (assuming there are only two labels).

Your rule is:

1. If p < 0.5 (threshold) then obs is labeled to class "one"
2. If p >= 0.5 (threshold) then obs is labeled to class "zero"

With ROC curve, you basically have a "high" number of thresholds, say going from 0.01 to 0.99. For each model.

So for each model, you will assign each observation to the respective class, using for t = 0.01, then t = 0.02, and so on till t = 0.99. (t = threshold)