How to calculate Accuracy, Precision, Recall and F1 score based on predict_proba matrix?

Question

I found this link that defines Accuracy, Precision, Recall and F1 score as:

Accuracy: the percentage of texts that were predicted with the correct tag.

Precision: the percentage of examples the classifier got right out of the total number of examples that it predicted for a given tag.

Recall: the percentage of examples the classifier predicted for a given tag out of the total number of examples it should have predicted for that given tag.

F1 Score: the harmonic mean of precision and recall.

Following this question of mine, my MultinomialNB classifier calculated the predict_proba matrix for the test set (with 14 samples) as follows:

0.192995    0.0996929   0.173688    0.136715    0.126616    0.133012    0.137282
0.174185    0.109345    0.169467    0.144389    0.115021    0.132762    0.154831
0.14172     0.190075    0.125429    0.155343    0.122939    0.149733    0.114763
0.130958    0.2304      0.108793    0.174371    0.115698    0.122529    0.117251
0.139486    0.0938475   0.236573    0.133689    0.118372    0.165151    0.112881
0.135901    0.0845106   0.262501    0.127767    0.119785    0.166609    0.102926
0.136622    0.13782     0.119651    0.320522    0.0854596   0.0996346   0.100292
0.139607    0.181654    0.112189    0.259983    0.0920986   0.106649    0.107819
0.151441    0.0929748   0.155358    0.130407    0.208591    0.151803    0.109425
0.132648    0.122881    0.130545    0.126466    0.196319    0.142594    0.148548
0.135545    0.101456    0.177762    0.118609    0.120773    0.253616    0.0922385
0.132612    0.112645    0.111808    0.102153    0.113548    0.327516    0.0997178
0.111618    0.0859541   0.106807    0.116613    0.085918    0.0873931   0.405696
0.107745    0.0936872   0.0877116   0.122336    0.0902212   0.0909265   0.407373

1. The Answerer of my last question, said that although the predict_proba matrix elements are all less than 0.5, they may be useful in text labeling. But From the above definitions, I concluded that the Accuracy and Precision of the prediction is zero, since all of the predicted values are less than 0.5. Am I correct?

2. I'm not sure about the Recall and F1 score and how to calculate them.

3. How can I interpret the matrix and the model's usefulness?

---

Edit 1:

Using this answer I changed my predict_proba matrix above (named in the code as pred_prob ) with a shape of (14,7) to a matrix (named y_pred) with a shape of (7,1) and then used a one_hot_encoder function to convert it to a confusion matrix (named y_pred_one_hot) as follows:

y_pred = np.argmax(pred_prob, axis=1)

def one_hot_encode(actual, n_classes):

    if len(actual.shape) == 1:
        actual2 = np.zeros((actual.shape[0], n_classes))
        for i, val in enumerate(actual):
            actual2[i, val] = 1
        actual = actual2

    return actual

y_pred_one_hot = one_hot_encode(y_pred, n_classes=7)

Now y_pred_one_hot is:

1   0   0   0   0   0   0
1   0   0   0   0   0   0
0   1   0   0   0   0   0
0   1   0   0   0   0   0
0   0   1   0   0   0   0
0   0   1   0   0   0   0
0   0   0   1   0   0   0
0   0   0   1   0   0   0
0   0   0   0   1   0   0
0   0   0   0   1   0   0
0   0   0   0   0   1   0
0   0   0   0   0   1   0
0   0   0   0   0   0   1
0   0   0   0   0   0   1

Now is this y_pred_one_hot matrix, the confusion matrix?

Answer 1

To compute performance metrics like precision, recall and F1 score you need to compare two things with each other:

• the predictions of your model for your evaluation set (in what follows, I'll call them y_pred)
• the true classes of your evaluation set (in what follows, y_true).

From what you write, you have obtained just the predictions of your model, and that's what you have in y_pred. You have constructed y_pred so that each of its components is equal to the class that is assigned the maximum probability by your model. All fine here!

The key ingredient that you are missing is the array of true classes (in your question, you called them "tags") associated to your evaluation examples. You totally need this information to understand if the predictions of your model are correct (or not). You should be able to construct an array y_true containing the true classes/tags of your examples from...knowing the actual ones. For example, if your 1st text belongs to class 3, your 2nd text belongs to class 1, your third text belongs to class 2, your y_true will be an array like

y_true = np.array([3, 1, 2, # ... the rest of components ])

Now, to compute accuracy, precision, and recall, you need to compare y_true and y_pred. If they coincide, congratulations: that means that your algorithm works perfectly on your evaluation set! In general though not all the components of y_pred will coincide with y_true. To quantify agreement/discrepancies you can use metrics like accuracy, precision, etc. You can code them yourself, but the scikit-learn library comes with functions for the purpose.

For instance you can easily compute accuracy, precision, recall and F1 score, even the confusion matrix for your problem with the code

from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# print accuracy
print("Accuracy: ", accuracy_score(y_true, y_pred))

# print precision, recall, F1-score per each class/tag
print(classification_report(y_true, y_pred))

# print confusion matrix, check documentation for sorting rows/columns
print(confusion_matrix(y_true, y_pred))

P.S. Note that in what I did I didn't use your y_pred_one_hot (which is not a confusion matrix!), and that the precision is not zero (it may be, but you have to compute it using y_true)!

P.S. Beware using predict_proba with Naive Bayes as output probabilities may not be calibrated.