Precision vs probability

Question

Say I have a model which predicts a class $C_i$ from an input $X$, with a probability of 0.95 i.e $P(C_i| X)=0.95$. That would mean that if we do this over and over, then 95/100 times we would be correct.

Having a model with a precision of 0.95 (for a threshold of, say, 0.7) would mean that 95/100 of the instances which have a score above 0.7 are classified as $C_i$ (vs. not $C_i$) are correct i.e $Pr(C_i|X) = 0.95$.

My question is; is there a difference between probability and precision e.g is there a difference in "how often we are right" when $P(C_i| X) = Pr(C_i |X)$?

I'm aware that a probabilities have to fulfill different criterias etc. (to call something a probability) but if we talk about probability in a model-perspective i.e how certain we are when we predict $C_i$.

The reason I ask is that I have a model which outputs some scores, which I want to transform/calibrate to something that describes how often we are right. We can try calibrate it to probabilities (e.g using sklearn) but some models are not easy to calibrate and might over-/underestimate the probabilities.

Instead of trying to calibrate it to probabilities, we can calculate the precision for different scores, and use that. That we can for all models, thus why would we chose the probability instead?

So what would the difference between having a mapper from "score to precision" vs "score to probability" be?

Answer 1

1. Precision is usually defined as a quality metric and describes (in binary classification) the fraction of correctly classified instances among all instances that it classified as True. There is a whole host of variations on the precision metric, extending to multiple labels or, for example, only pertaining to a subset of the data. Generally precision is defined as: $$\frac{\text{True Positives}}{\text{True Postives} + \text{False Positives}}$$
2. In binary classification, I would call the 'probability that a model predicts a label correctly' the accuracy. Usually calculated as $$\frac{\text{True Positives} + \text{True Negatives}}{\text{Number of samples}}$$
3. The 'probability that a model predicts one class correctly' is a little difficult to define as. What do you want to count with that? Do you count the True Negatives as correctly classified not to be of the positive class? Do you want to take that in your metric? In your example, you hypothesize to have model which predicts class $C_i$ correct with a probability of 0.95. What exactly does that mean? Is that the following? $$\frac{\text{True Positives}}{\text{Number of samples}}$$

Now, if you define precision and the 'probability that a model predicts a label correctly' as the same thing, then you can use them as the same thing. However, when talking about the 'probability that a model predicts a label correctly' and precision, I would think of the above definitions.

Now, all these numbers can be the same upon evaluation! However, they definitely do not need to be the same.