Should I try to predict the probability of being 1 or zero based on recall score?

Question

I built a random forest model to classify if a men's NCAA basketball game would go over or under the total. The model was given training data that specified over as 1 and under as 0. I then used a classification report to calculate the precision, recall, f1-score, and support. These were my results:

for a model like this when my goal is to accurately predict whether the test data will have an outcome of 1 or 0 should I use recall in my decision for choosing whether to try to predict if the outcome will be a 0 or 1?

Also I had my model using predict_proba and using the second value from the array return which would give me the probability of the test data outcome being a 1, but from this classification report it looks like I should be trying to predict the probability of the outcome being 0 as it is more descriptive when it comes to recall.

# Example:
# less accurate way

ynew = model.predict(Xnew)

results = ynew[0][1] # predict probability of Xnew being 1

# Example 2:
# more accurate way

ynew = model.predict(Xnew)

results = ynew[0][0] # predict probability of Xnew being 0

Is my assumption correct?

Answer 1

I have some bad news: your model is doing nothing useful.

• From the confusion matrix you can see that the model predicts class 0 for 68% of the instances and class 1 for 32%: (1446+1355)/4093=0.68 and (676+616)/4093=0.32.
• You can also see that when the true class is 0 the model predicts 68% of the instance as class 0 and 32% as class 1: 1446/2122=68% and 676/2122=32%. Almost same result when the true class is 1: 1355/1971=69% and 616/1971=31%.

This means that for any instance (no matter the true class) the model randomly predicts the label: class 0 with 68% chance, class 1 with 32% chance. You can also see this problem from the fact the sum of the top-left to bottom right diagonal (correctly classified cases) is 2062, almost the same as the bottom-left to top-right diagonal (errors) which is 2031. A good model would have a much larger difference between correct cases and incorrect cases.

This is why the accuracy is only 50.4%, barely more than a random baseline with two classes, and this is why the recall for each class corresponds exactly to the arbitrary probability that the model assigns to them.

should I use recall in my decision for choosing whether to try to predict if the outcome will be a 0 or 1?

This doesn't make sense: the recall is an evaluation score, it can only be calculated after the predictions have been made.

Also I had my model using predict_proba and using the second value from the array return which would give me the probability of the test data outcome being a 1, but from this classification report it looks like I should be trying to predict the probability of the outcome being 0 as it is more descriptive when it comes to recall.

There are at least two misunderstandings here:

• By itself, the fact that the model predicts class 0 more often than class 1 is neither good or bad. It would be completely wrong to choose to evaluate the performance using only the recall on the most frequent class: it's true that it's a higher value, but it's not at all a good performance indicator. For example, if one uses a model which always predicts class 0 (100% of the time) then by definition this classifier will have recall 100% for class 0, even though this model is pointless.
• In binary classification, the probability of predicting the two classes are mathematically bound together: $p(C=0)+p(C=1)=1$. So choosing to predict the outcome 0 instead of 1 doesn't change anything about what the model does, it would be exactly the same model with the same performance.

What I suggest you should try is to work on your features: normally the features should be indicators that the model uses to predict the class, but unfortunately your current features are terrible at their job. Assuming that the features "make sense" for the task, it should be possible to "present them" to the model so that it can actually use them efficiently. This is called feature engineering: it's often the hard part of the job but also the most likely to really make things work :)