I struggle to understand the mathematical motivation for the binary classification model calibration curve. Why do we assume that the predicted probabilities should be consistent with the proportion of 1's in the probability bin (# of 1's in the bin)/(total # of samples in the bin) ? It's obvious for Decision Tree where the (# of 1's in the bin)/(total # of samples in the bin) ratio is explicitly the model output, but how is this related to the other models? How to show that this ratio and predicted probabilities should be equal?

  • $\begingroup$ What class ratio are you assuming? That the output of a binary classification is binary? $\endgroup$
    – WBM
    Mar 19 '21 at 12:58
  • $\begingroup$ Thanks for pointing this out, I was incorrect when asked the question, now I've edited it. I mean the ratio of 1's to total number of samples in the probability bin $\endgroup$ Mar 19 '21 at 13:53
  • 1
    $\begingroup$ That is the empirical probability of the event in the bin. $\endgroup$ Mar 21 '21 at 22:30
  • $\begingroup$ I really can't embrace this idea, why should the probability, that we took the item with class 1 (p = #1/#total) from the bin, be equal the probability that the given item from the bin belongs to class 1. But maybe now I can after writtinig this... $\endgroup$ Mar 22 '21 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.