0
$\begingroup$

I'm working on a classification problem in order to predict among 50 different classes. I'm using a Random Forest classifier and I'm using the predict_proba method to the fitted RandomForestClassifier in order to know the estimated probability of the predicted class.

Then, for each class, I calculate this estimated probability mean m. This mean could be interpreted as "On average, for this predicted class, the RF classifier estimates that it has m% chance that the prediction is correct". On the other side, the recall is the percentage of correct predictions for each class (fixed ground truth).

If I'm right, and if it's correctly constructed, this mean should be close the class recall. But is it ? Is this mean a consistent estimator of the class recall ? Should it ?

$\endgroup$

2 Answers 2

2
$\begingroup$

Unfortunately this doesn't work. To understand why, let's think about the perfect classifier and the maximally wrong classifier.

The perfect classifier has 100% recall on any given class. If there are 5 uniformly distributed classes, the average probability assigned to each of those classes by the classifier (over some set of data) will be 20% (because it will assign that class 100% when it is the true class, which is 20% of the time, and 0% at all other times).

The maximally wrong classifier, which has 0% recall on every class, still assigns probability to wrong classes. Thus its average probability assigned to a class is > 0% even though recall is strictly 0%.

Your classifier probably isn't perfect or maximally wrong, but hopefully this helps you have an intuition for why recall and average class probability are unrelated.

$\endgroup$
1
  • $\begingroup$ Hi Nicholas, thanks for your reply. In the first case, I'm not sure that the classifier will predict a 20% probability with the predict_proba function for the predicted class even with uniformly distributed classes. This will depend on the features and how it is able to separate the classes depending on the features. But I can understand what you mean with your second example. Thank you ! $\endgroup$
    – Jerome X.
    Sep 21, 2023 at 14:41
1
$\begingroup$

This does not make sense.

When you predict the probability, there is not the sense of being right or wrong like there is when you predict a category. Therefore, there is not a notion of recall, let alone consistent estimation of the recall.

What might be of interest to you is the calibration of your probability predictions: whether or not a predicted probability of $p$ corresponds with the event happening $p$ proportion of times. Especially for multi-class problems, assessing this seems to be an open problem. Note that random forests are not known for their calibration.

$\endgroup$
1
  • $\begingroup$ Yes that's exactly what I would like ! A probability p which corresponds to the event happening p proportion of times. I'll check on the links you provide ! Thanks for you reply ! $\endgroup$
    – Jerome X.
    Sep 21, 2023 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.