# Comparing probability models for alignment

I have a probability model which predicts a probability for a binary classification problem. I am interested in how well the predicted probability aligns with the true probability. For instance, you could have two models $$p_1(x)$$ and $$p_2(x)$$ with the same ROC curve. The thresholds for $$p_1(x)$$ match the true long-run probability of classifying that object where as the thresholds for $$p_2(x)$$ are some distorted probability $$\frac{1}{1 + e^{-4 p_l}}$$ where $$p_l$$ is the long-run probability for an instance $$x$$. $$p_2(x)$$ would not be very well aligned.

I have collected a variety of data with my predicted probability value as well as the underlying truth:

p v
0.2 0
0.2 0
0.2 1
0.2 0
0.2 0
0.8 1
0.8 0
0.8 1
0.8 1
1.0 1
0.1 1

Is there a standard way to measure / discuss this? Ideally if these probability values were discretized, the mean of each bucket could be taken and compared to the values $$p$$. I was thinking I could take the logits of $$p$$ and fit a logistic regression to $$v$$. This should provide a measure of "alignment."