# How to compute precision and accuracy of a sequence that is not strictly binary?

Given a predicted sequence and actual sequence I want to compute it's precision and accuracy, for example: Note that these sequences will only contain 0, 1 or -1

predicted sequence: -1,0,1,1,-1,0,1,1,0,-1

actual sequence: -1,1,0,1,-1,1,0,1,0,-1

I know that precision is computed using this tp/tp+fp and accuracy is computed using tp + tn /tp + tn + fp + fn. But because I have -1 in it I am unsure how I would compute true positives? My understanding that a true positive is if I predicted a 1 and it's corresponding actual value is a 1. A walk through of the computation for precision and accuracy would help.

• Accuracy is possible?Accuracy = what fraction of all predictions did we get right? Most metrics (except accuracy) are generally analysed as multiple 1-vs-many Mar 12 '18 at 0:17

Welcome to the Site!

We know that this problem is Multi-Class Classification Problem.

To get a confusion matrix for the same you can use the following command:

from mlxtend.evaluate import confusion_matrix

#import the required packages
from mlxtend.evaluate import confusion_matrix
from mlxtend.evaluate import plot_confusion_matrix

#Actual Target Values
y_target =    [-1,1,0,1,-1,1,0,1,0,-1]
#Predicted Values
y_predicted = [-1,0,1,1,-1,0,1,1,0,-1]

#creation of confusion matrix
cm = confusion_matrix(y_target=y_target,
y_predicted=y_predicted,
binary=False)
#to print the calculated values  of Confusion Matrix
cm


Outcome:

array([[3, 0, 0],
[0, 1, 2],
[0, 2, 2]])


For visualizing the cm you can use the following command:

fig, ax = plot_confusion_matrix(conf_mat=cm)
plt.show()


You can go through this Link for better understanding of mlextend.

You can get the Precision and Accuracy values by using the following formulas:

$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{kji}}$

$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ijk}}$

Go through these Link-1,Link-2 for better understanding on how to compute the same, in the Link-3 is GitHub link which explains on how they implemented for a 1-D array, looking at that you can try expanding it for your outcome.

• Thanks for this. Using sklearns precision_score and accuracy_score functions, does it capture this multi class aspect? Mar 12 '18 at 2:02
• yeah that should do the thing as well, this Link has the implementation. If you are just looking for calculating these values. Mar 12 '18 at 2:04
• If you want to visualize the CM then you can use the above and show it in a proper format for better understanding Mar 12 '18 at 2:34
• What is the CM? Mar 12 '18 at 4:34
• I've appended the answer with the visualization as well, have a look Mar 12 '18 at 5:45

Disclaimer,

Hello,You can also use PyCM which is a python module that evaluate the performance of your classifier using not only just precision and recall but also a wide variety of metrics.

You can use it via following commands

>>> from pycm import ConfusionMatrix
>>> cm1=ConfusionMatrix(y_target,y_predicted)
>>> print(cm1)
Predict          -1    0     1
Actual
-1               3     0     0
0                0     1     2
1                0     2     2
Overall Statistics :
95% CI                                                           (0.29636,0.90364)
AUNP                                                             0.69048
AUNU                                                             0.70238
Bennett S                                                        0.4
CBA                                                              0.61111
Chi-Squared                                                      10.27778
Chi-Squared DF                                                   4
Conditional Entropy                                              0.67549
Cramer V                                                         0.71686
Cross Entropy                                                    1.57095
Gwet AC1                                                         0.40299
Hamming Loss                                                     0.4
Joint Entropy                                                    2.24644
KL Divergence                                                    0.0
Kappa                                                            0.39394
Kappa 95% CI                                                     (-0.06612,0.854)
Kappa No Prevalence                                              0.2
Kappa Standard Error                                             0.23473
Kappa Unbiased                                                   0.39394
Lambda A                                                         0.5
Lambda B                                                         0.5
Mutual Information                                               0.89546
NIR                                                              0.4
Overall ACC                                                      0.6
Overall CEN                                                      0.3585
Overall J                                                        (1.53333,0.51111)
Overall MCC                                                      0.39394
Overall MCEN                                                     0.41527
Overall RACC                                                     0.34
Overall RACCU                                                    0.34
P-Value                                                          0.16624
PPV Macro                                                        0.61111
PPV Micro                                                        0.6
Phi-Squared                                                      1.02778
RCI                                                              0.57001
RR                                                               3.33333
Reference Entropy                                                1.57095
Response Entropy                                                 1.57095
SOA1(Landis & Koch)                                              Fair
SOA2(Fleiss)                                                     Poor
SOA3(Altman)                                                     Fair
SOA4(Cicchetti)                                                  Poor
Scott PI                                                         0.39394
Standard Error                                                   0.15492
TPR Macro                                                        0.61111
TPR Micro                                                        0.6
Zero-one Loss                                                    4
Class Statistics :
Classes                                                          -1                      0                       1
ACC(Accuracy)                                                    1.0                     0.6                     0.6
AUC(Area under the roc curve)                                    1.0                     0.52381                 0.58333
AUCI(Auc value interpretation)                                   Excellent               Poor                    Poor
BM(Informedness or bookmaker informedness)                       1.0                     0.04762                 0.16667
CEN(Confusion entropy)                                           0                       0.52832                 0.5
DOR(Diagnostic odds ratio)                                       None                    1.25                    2.0
DP(Discriminant power)                                           None                    0.05343                 0.16597
DPI(Discriminant power interpretation)                           None                    Poor                    Poor
ERR(Error rate)                                                  0.0                     0.4                     0.4
F0.5(F0.5 score)                                                 1.0                     0.33333                 0.5
F1(F1 score - harmonic mean of precision and sensitivity)        1.0                     0.33333                 0.5
F2(F2 score)                                                     1.0                     0.33333                 0.5
FDR(False discovery rate)                                        0.0                     0.66667                 0.5
FN(False negative/miss/type 2 error)                             0                       2                       2
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5
FOR(False omission rate)                                         0.0                     0.28571                 0.33333
FP(False positive/type 1 error/false alarm)                      0                       2                       2
FPR(Fall-out or false positive rate)                             0.0                     0.28571                 0.33333
G(G-measure geometric mean of precision and sensitivity)         1.0                     0.33333                 0.5
IS(Information score)                                            1.73697                 0.152                   0.32193
J(Jaccard index)                                                 1.0                     0.2                     0.33333
MCC(Matthews correlation coefficient)                            1.0                     0.04762                 0.16667
MCEN(Modified confusion entropy)                                 0                       0.52877                 0.52832
MK(Markedness)                                                   1.0                     0.04762                 0.16667
N(Condition negative)                                            7                       7                       6
NLR(Negative likelihood ratio)                                   0.0                     0.93333                 0.75
NPV(Negative predictive value)                                   1.0                     0.71429                 0.66667
P(Condition positive or support)                                 3                       3                       4
PLR(Positive likelihood ratio)                                   None                    1.16667                 1.5
PLRI(Positive likelihood ratio interpretation)                   None                    Poor                    Poor
POP(Population)                                                  10                      10                      10
PPV(Precision or positive predictive value)                      1.0                     0.33333                 0.5
PRE(Prevalence)                                                  0.3                     0.3                     0.4
RACC(Random accuracy)                                            0.09                    0.09                    0.16
RACCU(Random accuracy unbiased)                                  0.09                    0.09                    0.16
TN(True negative/correct rejection)                              7                       5                       4
TNR(Specificity or true negative rate)                           1.0                     0.71429                 0.66667
TON(Test outcome negative)                                       7                       7                       6
TOP(Test outcome positive)                                       3                       3                       4
TP(True positive/hit)                                            3                       1                       2
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5
Y(Youden index)                                                  1.0                     0.04762                 0.16667
dInd(Distance index)                                             0.0                     0.72531                 0.60093
sInd(Similarity index)                                           1.0                     0.48713                 0.57508