2
$\begingroup$

I've the probability scores of positive class of two models. The frequency distribution of those probability scores(thresholds) are like this

Model #1 enter image description here

Model #2 enter image description here

It's a binary classification problem. And the thresholds obtained are from the test data. The stats in the confusion matrix are nearly same for two models. Can we say the Model #2 is doing well in separating classes? What other conclusion we can draw from the frequency of the thresholds.

$\endgroup$
0
$\begingroup$

I'd say the model 1 is performing really well. If you can use different colors while plotting for positive and negative classes, then you should be able to see the difference. When you are trying to do binary classification, the distributions of negative and positive classes should be dipping in the mid region where as Model 2 is the opposite.

$\endgroup$
  • $\begingroup$ But, I think no need to color it, as we know the observation >0.5 are positively predicted, and rest all are negatively predicted. Can you please explain more on why "dipped in the mid region" is expected? $\endgroup$ – Venkatesh Gandi Jan 16 at 16:23
  • $\begingroup$ You don't have to colour it but it helps to differentiate between classes. Dipping in the mid region is good to have because all the negative classes should be skewed to the left and the positive classes should be skewed to the right which means that the model is able to classify well. $\endgroup$ – Danny Jan 16 at 16:29
  • $\begingroup$ Ok, I'm thinking in a diagnostic manner, in model #1 when we change the threshold from 0.5 to 0.3 nothing much changed in confusion matrix(as it less densed around 0.5). So for the model which has good class separation, the frequency of threshold will look like the one in the model #1. Am I correct? $\endgroup$ – Venkatesh Gandi Jan 16 at 16:36
  • $\begingroup$ Yes absolutely. $\endgroup$ – Danny Jan 16 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.