2
$\begingroup$

I have a Random Forest Classifier (trained with sklearn) modeling a binary data set. Here's what the configuration looks like (I've tuned it for precision intentionally):

rf_estimator = RandomForestClassifier(
    class_weight="balanced",
    random_state=1, n_jobs=-1,
    max_depth=20, max_features=0.4, n_estimators=1000, min_samples_leaf=20
)

And here's what the results of testing with 30% of the original data set looks like:

[[1051  788]
 [1438 7230]]
   Precision    Recall  Accuracy
0   0.901721  0.834102  0.788141

Now what I'm trying to understand is how (if at all) I should be using these scores when interpreting the the probability results of predicting new data. So for example, let's say I give the model a new piece of data to classify and it returns:

(0.3, 0.7)

Is there only a 90% chance that the 0.7 is correct? Or does the 0.7 take into account the 90%? In other words, can I take the probabilities at face value (i.e., "we're 70% confident that this is a Class 1") or do I need to do some extra maths given what I know about my models precision/accuracy?

$\endgroup$

1 Answer 1

2
$\begingroup$

So for example, let's say I give the model a new piece of data to classify and it returns: (0.3, 0.7)

Is there only a 90% chance that the 0.7 is correct?

The 90% precision means that when the model predicts y=1, then it tends to be right. This is a measure of the model's reliability when it comes to the positive class. Its positive predictions are correct 90% of the time.

It doesn't mean that the value of 0.7 is correct, if that's what you were asking. Rather, since the model has predicted y=1 (presuming the decision threshold is below 0.7), you could interpret it as a 90% chance of it being the correct class. It's an averaged metric, so I don't think it should be taken too literally for individual samples.

Or does the 0.7 take into account the 90%?

The 90% is a reliability measure of the classifier's positive predictions. The 0.7 is a measure of the model's confidence about its output. They could interact, but I don't think there's necessarily a relationship.

For example, a classifier could marginally predict positive (0.51) for every positive case, resulting in a classifier that's under-confident but nevertheless reliably predicts the positive cases. The reverse is also true (and perhaps more frequent); a classifier that's sure about its prediction (0.99) but is always wrong. This is just to demonstrate the point that the classifier's confidence scores and precision are separate entities. I don't think you can use one to glean much about the other.

$\endgroup$
4
  • $\begingroup$ 'Usually, a classifier's output can't be directly interpreted as a probability.' Highly disagree. In many cases it should be interpreted as a probability. For instance, logistic regression can be interpreted as probabilistic classification and it is designed to estimate the (conditional) probabilities of the outcomes. $\endgroup$ Commented Jun 16 at 13:24
  • $\begingroup$ I agree with your point on logistic regression. By "probability" I meant actual/real-world proportions of the positive class, for which a further calibration step often helps by correcting for over- or under-confidence in the model's output. $\endgroup$ Commented Jun 16 at 14:57
  • $\begingroup$ The fact that calibration often helps is irrelevant for this. Many probabilistic methods/classifiers estimate exactly the 'real-world proportions of the positive class' you mention. $\endgroup$ Commented Jun 17 at 11:41
  • $\begingroup$ Yes, I think calibration is tangential to the OP's question, which is why I had removed that part of the answer. About the classifiers estimating real-world proportions...their estimates can be strongly biased by modelling assumptions though, right? Estimates could be systematically pushed out towards 0/1, or pulled in towards 0.5, depending on the model. Calibration is for mapping those values onto actual observed probabilities, which is useful but not what the OP was asking about. That's my impression of the discussion at this link. $\endgroup$ Commented Jun 17 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.