2
$\begingroup$

As the title reads what is the difference? Plotting the ROC w.r.t probability scores gives the stair cased version.

But in my opinion I find that using binary decision is better because the ROC curve is just a summary of the different confusion matrices w.r.t different thresholds. And the confusion matrix shows the binary decisions.

Is their an advantage or certain situation where plotting the ROC curve with the prob.scores is better than the binary decisions?

Improve this question
$\endgroup$
2
  • $\begingroup$ Please read about proper scoring rules. There is a good answer here, and the links within that answer (and the links within those, especially Harrell's blog) are worth reading: stats.stackexchange.com/questions/464636/…. $\endgroup$
    – Dave
    Sep 14, 2020 at 15:02
  • $\begingroup$ I don't see how this is about proper scoring rules in the sense that is described in the link. Scikit user guide under chapter 3.3 - Metrics and scoring: quantifying the quality of prediction, states "This function (roc_curve) requires the true binary value and the target scores, which can either be probability estimates of the positive class, confidence values, or binary decisions " Hence why I was wondering what the difference and the advantages are. $\endgroup$
    – John
    Sep 14, 2020 at 15:37

1 Answer 1

Reset to default
2
$\begingroup$

How do you go from probabilities to binary decisions? The default choice is to use a threshold of 0.5 but maybe a threshold of 0.3 or 0.7 would have given better results (depending on you metric). The ROC curve gives you more information as it allows to see the results for each probability threshold.

Usually you set some metric to optimize (F1 score for example) and you set the threshold based on this metric. Then you plot the confusion matrix and any other metric that is useful to you

Improve this answer
$\endgroup$
9
  • $\begingroup$ I understand that the probabilty is what determines the binary classification but as I said what is the difference? The ROC gives result based on different thresholds, as you said, but then it shouldnt matter if I choose the curve based on the predicted probabilties or the binary classes. $\endgroup$
    – John
    Sep 14, 2020 at 13:10
  • $\begingroup$ Colud you clarify a bit more "choose the curve" and "binary classes" ? $\endgroup$
    – mprouveur
    Sep 14, 2020 at 13:21
  • $\begingroup$ Binary classes = when the output is 1 or 0. I use a logistic regression model which the output can be in form av decision function, probabilites or predictions (i.e. 1 or 0). Hence, I was wondering what the difference is when I plot the ROC in terms of probabilites or the output in terms of predicitons (the binary classes). $\endgroup$
    – John
    Sep 14, 2020 at 14:40
  • $\begingroup$ Could you post some code of what you are actually comparing? ROC curves don't have a meaning to me if you give it your thresholded predictions instead of your probabilities. To my knowledge ROC curves are plotted using the various thresholds you could set on your probability ouput. If you pass it the thresholded values nothing should vary (or maybe by default you would have 3 points : one if you set everything to 0, one that actually represent the score of your model with the default threshold of 0.5, and one if you set everything to 1) $\endgroup$
    – mprouveur
    Sep 15, 2020 at 9:11
  • $\begingroup$ Okey, so to make it simple, why wouldnt this work:1. vary the threshold at which you'd predict either a 0 or 1 2.At different thresholds compute the true positive rate (TPR) and false positive rate (FPR) Plot TPR vs FPR 3. Plot the graph. $\endgroup$
    – John
    Sep 15, 2020 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.