1
$\begingroup$

I want to plot ROC curves using R. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a label vector. The column names of prediction columns are ccs,badaI,badaII and the column name of label vector is value. I am using ROCRlibrary for this as:

library(ROCR)
pred1 <- prediction(df$ccs,df$value)
roc <- performance(pred1,"tpr","fpr");
pred2 <- prediction(df$badaI,df$value)
roc2  <- performance(pred2,"tpr","fpr")
pred3 <- prediction(df$badaII,df$value)
roc3 <- performance(pred3,"tpr","fpr")
auc<- performance(pred,"auc")
auc = round(unlist([email protected]),2)
auc2<- performance(pred2,"auc")
auc2 = round(unlist([email protected]),2)
auc3<- performance(pred3,"auc")
auc3 = round(unlist([email protected]),2)
plot(roc,col="black",lty=1, lwd=4, cex.lab=1.5,axt="n")
axis(1,cex.axis=1.0);axis(2,cex.axis=1.0)
plot(roc2, add=TRUE,col="black",lty=3, lwd=4)
plot(roc3, add=TRUE,col="black",lty=2, lwd=4)
abline(0,1,col="gray60")
legend(0.3,0.30,c(paste0("CCS, ","AUC = ",auc),paste0("BADAI, ","AUC = ",auc2),paste0("BADAII, ","AUC = ",auc3)),
      lty=c(1,3,2), col=c('black','black','black'), lwd=4,cex=1.4,bty="n")

While using the above code,I am getting following plot: enter image description here

I have doubt as: While looking at the data, It is obvious that ccs and badaII should have higher AUC values than badaI, but the results are somehow opposite. Can anyone help me in understanding why it is behaving like this? The dput of the data used, df is: 

structure(list(ccs = c(0.16, 0.04, 0.18, 0.09, 0.14, 0.14, 0.04, 
0.04, 0.08, 0.76, 0.03, 0.03, 0.68, 0.06, 0.83, 0.15, 0.07, 0.02, 
0.93, 0.22, 0.28, 0.11, 0.05, 0.01, 0.17, 0.15, 1, 0.13, 0.23, 
0.44, 1), badaI = c(0.61, 0.11, 0.53, 0.79, 0.75, 0.82, 0.57, 
0.67, 0.4, 0.95, 0.49, 0.61, 0.97, 0.52, 0.98, 0.7, 0.03, 0.18, 
0.85, 0.94, 0.9, 0.77, 0, 0.37, 0.47, 0.88, 0.99, 0.55, 0.86, 
0.96, 0.99), badaII = c(0.32, 0, 0.27, 0.12, 0.33, 0.12, 0.56, 
0, 0.32, 0.18, 0.18, 0.11, 0.18, 0.54, 0.37, 0.33, 1, 0.39, 0.29, 
0.11, 0.32, 0.53, 0.25, 0.21, 0.15, 0.16, 0.85, 0.31, 0.44, 1, 
1), value = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1)), .Names = c("ccs", 
"badaI", "badaII", "value"), row.names = c(NA, -31L), class = "data.frame")

UPDATE I use below figure to further explain my intuition. This figure is drawn using the same predictor values. enter image description here

According to value column, observation with numbers 15, 27, 30, 31 have labels as 1 and the remaining observations have 0 value. While looking at above figure it is clear that CCS or badaIIare best in discriminating the difference between 0 and 1 as compared to badaI, which always provide higher predictor values.In other words, I mean with badaI, it is difficult to predict 1 as its values are higher for both 0 and 1.

I am not able to correlate my intuition with the ROC plot. @TBSRounder, I understood what you have mentioned, but I need to support the above figure with the ROC plot, which I find disappointing. Can anyone help me to correlate the above figure with the ROC plot?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The results you posted are correct, I did a quick check with library(pROC) and got the same thing. The important feature for AUC/ROC is that the cutpoint of calling a sample "1" or "0" is not set at 0.5. For each of your predictors, the range of the numbers do not matter at all. What does matter is how often higher numbers are associated with true positive labels. Example:

Method 1:[1, 3, 5, 6, 9, 14] (predictor)
Method 1:[0, 0, 1, 0 ,1 , 1] (true label)
AUC: 0.8889

Method 2:[[0.01, 0.02, 200 , 250, 300, 1000000] (predictor)
Method 2:[0, 0, 1, 0 ,1 , 1] (true label)
AUC: 0.8889

The AUC measure is just a measure of how often true "1" samples have a higher number than true "0" samples. The predictor does not have to even be bounded by [0,1], it can be any range.

Your plot would be more informative if it was sorted. Consider the below sorted plot (crude, sorry) for each of the three methods, with the "1" labels highlighted in blue. badaI clearly has the closest grouping of "1"s towards the higher values.

Sorted Values by Method

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your effort. I understood you answer. I have updated my question with the UPDATE section. Can you please read it again? $\endgroup$
    – Haroon Lone
    Commented May 3, 2016 at 13:18
  • $\begingroup$ @Haroon I updated my answer, sorting your plots and highlight the true "1"s would represent the AUC score better. $\endgroup$
    – TBSRounder
    Commented May 3, 2016 at 13:42
  • $\begingroup$ Thanks. I want to understand it further. Does this mean that ROC curve do not consider False positive Rate (FPR)? If I consider 0.75 as threshold then badaI has highest number of false positives. Similarly, at other thresholds badaI have highest FPR as compared to other two approaches. How should I show this? $\endgroup$
    – Haroon Lone
    Commented May 3, 2016 at 15:38
  • 1
    $\begingroup$ The ROC curve considers every threshold between the smallest value and the largest value. If you have 31 probabilities, then you can think of 31 confusion matrices, one for each cutpoint. Those confusion matrices each have their own TPR and FPR. If you plotted TPR and FPR for each of those tables as you increased the threshold, you will get the ROC curve. ROC's always start at (0,0) because at the extreme case, it classifies everything as negative (threshold is the max(p)), so there are no TP or FP. Similar intuition for min(p) where the threshold calls everything positive, end at (1,1) $\endgroup$
    – TBSRounder
    Commented May 3, 2016 at 16:31
  • $\begingroup$ @HaroonRashid: When using AUROC, you will not get insight by choosing a single threshold for comparison. You have to ignore the value output entirely, and consider the predicted ranking that it produces in your test set. For intuition, consider finding the optimal threshold for each model separately, and see what the TPR/FPR is for that. In this case, badaI is strictly better than the other models, when you want to minimise FPR for any given TPR. $\endgroup$
    – Neil Slater
    Commented May 4, 2016 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.