Is probabilities mean of predicted class (RandomForest) a consistent estimator of class recall?

Question

I'm working on a classification problem in order to predict among 50 different classes. I'm using a Random Forest classifier and I'm using the predict_proba method to the fitted RandomForestClassifier in order to know the estimated probability of the predicted class.

Then, for each class, I calculate this estimated probability mean m. This mean could be interpreted as "On average, for this predicted class, the RF classifier estimates that it has m% chance that the prediction is correct". On the other side, the recall is the percentage of correct predictions for each class (fixed ground truth).

If I'm right, and if it's correctly constructed, this mean should be close the class recall. But is it ? Is this mean a consistent estimator of the class recall ? Should it ?

Answer 1

Unfortunately this doesn't work. To understand why, let's think about the perfect classifier and the maximally wrong classifier.

The perfect classifier has 100% recall on any given class. If there are 5 uniformly distributed classes, the average probability assigned to each of those classes by the classifier (over some set of data) will be 20% (because it will assign that class 100% when it is the true class, which is 20% of the time, and 0% at all other times).

The maximally wrong classifier, which has 0% recall on every class, still assigns probability to wrong classes. Thus its average probability assigned to a class is > 0% even though recall is strictly 0%.

Your classifier probably isn't perfect or maximally wrong, but hopefully this helps you have an intuition for why recall and average class probability are unrelated.

Answer 2

This does not make sense.

When you predict the probability, there is not the sense of being right or wrong like there is when you predict a category. Therefore, there is not a notion of recall, let alone consistent estimation of the recall.

What might be of interest to you is the calibration of your probability predictions: whether or not a predicted probability of $p$ corresponds with the event happening $p$ proportion of times. Especially for multi-class problems, assessing this seems to be an open problem. Note that random forests are not known for their calibration.