I have a probability model which predicts a probability for a binary classification problem. I am interested in how well the predicted probability aligns with the true probability. For instance, you could have two models $p_1(x)$ and $p_2(x)$ with the same ROC curve. The thresholds for $p_1(x)$ match the true long-run probability of classifying that object where as the thresholds for $p_2(x)$ are some distorted probability $\frac{1}{1 + e^{-4 p_l}}$ where $p_l$ is the long-run probability for an instance $x$. $p_2(x)$ would not be very well aligned.
I have collected a variety of data with my predicted probability value as well as the underlying truth:
| p | v |
|---|---|
| 0.2 | 0 |
| 0.2 | 0 |
| 0.2 | 1 |
| 0.2 | 0 |
| 0.2 | 0 |
| 0.8 | 1 |
| 0.8 | 0 |
| 0.8 | 1 |
| 0.8 | 1 |
| 1.0 | 1 |
| 0.1 | 1 |
Is there a standard way to measure / discuss this? Ideally if these probability values were discretized, the mean of each bucket could be taken and compared to the values $p$. I was thinking I could take the logits of $p$ and fit a logistic regression to $v$. This should provide a measure of "alignment."
