I struggle to understand the mathematical motivation for the binary classification model calibration curve. Why do we assume that the predicted probabilities should be consistent with the proportion of 1's in the probability bin (# of 1's in the bin)/(total # of samples in the bin) ? It's obvious for Decision Tree where the (# of 1's in the bin)/(total # of samples in the bin) ratio is explicitly the model output, but how is this related to the other models? How to show that this ratio and predicted probabilities should be equal?

  • $\begingroup$ What class ratio are you assuming? That the output of a binary classification is binary? $\endgroup$
    – WBM
    Commented Mar 19, 2021 at 12:58
  • $\begingroup$ Thanks for pointing this out, I was incorrect when asked the question, now I've edited it. I mean the ratio of 1's to total number of samples in the probability bin $\endgroup$ Commented Mar 19, 2021 at 13:53
  • 1
    $\begingroup$ That is the empirical probability of the event in the bin. $\endgroup$
    – Ben Reiniger
    Commented Mar 21, 2021 at 22:30
  • $\begingroup$ I really can't embrace this idea, why should the probability, that we took the item with class 1 (p = #1/#total) from the bin, be equal the probability that the given item from the bin belongs to class 1. But maybe now I can after writtinig this... $\endgroup$ Commented Mar 22, 2021 at 9:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.