1
$\begingroup$

Given a predicted sequence and actual sequence I want to compute it's precision and accuracy, for example: Note that these sequences will only contain 0, 1 or -1

predicted sequence: -1,0,1,1,-1,0,1,1,0,-1

actual sequence: -1,1,0,1,-1,1,0,1,0,-1

I know that precision is computed using this tp/tp+fp and accuracy is computed using tp + tn /tp + tn + fp + fn. But because I have -1 in it I am unsure how I would compute true positives? My understanding that a true positive is if I predicted a 1 and it's corresponding actual value is a 1. A walk through of the computation for precision and accuracy would help.

$\endgroup$
1
  • $\begingroup$ Accuracy is possible?Accuracy = what fraction of all predictions did we get right? Most metrics (except accuracy) are generally analysed as multiple 1-vs-many $\endgroup$
    – Aditya
    Mar 12 '18 at 0:17
3
$\begingroup$

Welcome to the Site!

We know that this problem is Multi-Class Classification Problem.

To get a confusion matrix for the same you can use the following command:

from mlxtend.evaluate import confusion_matrix

#import the required packages
from mlxtend.evaluate import confusion_matrix 
from mlxtend.evaluate import plot_confusion_matrix

#Actual Target Values
y_target =    [-1,1,0,1,-1,1,0,1,0,-1]
#Predicted Values
y_predicted = [-1,0,1,1,-1,0,1,1,0,-1]

#creation of confusion matrix
cm = confusion_matrix(y_target=y_target, 
                      y_predicted=y_predicted, 
                      binary=False)
#to print the calculated values  of Confusion Matrix
cm

Outcome:

array([[3, 0, 0],
   [0, 1, 2],
   [0, 2, 2]])

For visualizing the cm you can use the following command:

fig, ax = plot_confusion_matrix(conf_mat=cm)
plt.show()

OutcomeViz

You can go through this Link for better understanding of mlextend.

You can get the Precision and Accuracy values by using the following formulas:

$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{kji}}$

$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ijk}}$

Go through these Link-1,Link-2 for better understanding on how to compute the same, in the Link-3 is GitHub link which explains on how they implemented for a 1-D array, looking at that you can try expanding it for your outcome.

$\endgroup$
7
  • $\begingroup$ Thanks for this. Using sklearns precision_score and accuracy_score functions, does it capture this multi class aspect? $\endgroup$ Mar 12 '18 at 2:02
  • $\begingroup$ yeah that should do the thing as well, this Link has the implementation. If you are just looking for calculating these values. $\endgroup$
    – Toros91
    Mar 12 '18 at 2:04
  • $\begingroup$ If you want to visualize the CM then you can use the above and show it in a proper format for better understanding $\endgroup$
    – Toros91
    Mar 12 '18 at 2:34
  • $\begingroup$ What is the CM? $\endgroup$ Mar 12 '18 at 4:34
  • 1
    $\begingroup$ I've appended the answer with the visualization as well, have a look $\endgroup$
    – Toros91
    Mar 12 '18 at 5:45
0
$\begingroup$

Disclaimer,

Hello,You can also use PyCM which is a python module that evaluate the performance of your classifier using not only just precision and recall but also a wide variety of metrics.

You can use it via following commands

>>> from pycm import ConfusionMatrix
>>> cm1=ConfusionMatrix(y_target,y_predicted)
>>> print(cm1)
Predict          -1    0     1     
Actual
-1               3     0     0     
0                0     1     2     
1                0     2     2     
Overall Statistics : 
95% CI                                                           (0.29636,0.90364)
AUNP                                                             0.69048
AUNU                                                             0.70238
Bennett S                                                        0.4
CBA                                                              0.61111
Chi-Squared                                                      10.27778
Chi-Squared DF                                                   4
Conditional Entropy                                              0.67549
Cramer V                                                         0.71686
Cross Entropy                                                    1.57095
Gwet AC1                                                         0.40299
Hamming Loss                                                     0.4
Joint Entropy                                                    2.24644
KL Divergence                                                    0.0
Kappa                                                            0.39394
Kappa 95% CI                                                     (-0.06612,0.854)
Kappa No Prevalence                                              0.2
Kappa Standard Error                                             0.23473
Kappa Unbiased                                                   0.39394
Lambda A                                                         0.5
Lambda B                                                         0.5
Mutual Information                                               0.89546
NIR                                                              0.4
Overall ACC                                                      0.6
Overall CEN                                                      0.3585
Overall J                                                        (1.53333,0.51111)
Overall MCC                                                      0.39394
Overall MCEN                                                     0.41527
Overall RACC                                                     0.34
Overall RACCU                                                    0.34
P-Value                                                          0.16624
PPV Macro                                                        0.61111
PPV Micro                                                        0.6
Phi-Squared                                                      1.02778
RCI                                                              0.57001
RR                                                               3.33333
Reference Entropy                                                1.57095
Response Entropy                                                 1.57095
SOA1(Landis & Koch)                                              Fair
SOA2(Fleiss)                                                     Poor
SOA3(Altman)                                                     Fair
SOA4(Cicchetti)                                                  Poor
Scott PI                                                         0.39394
Standard Error                                                   0.15492
TPR Macro                                                        0.61111
TPR Micro                                                        0.6
Zero-one Loss                                                    4
Class Statistics :
Classes                                                          -1                      0                       1                       
ACC(Accuracy)                                                    1.0                     0.6                     0.6                     
AUC(Area under the roc curve)                                    1.0                     0.52381                 0.58333                 
AUCI(Auc value interpretation)                                   Excellent               Poor                    Poor                    
BM(Informedness or bookmaker informedness)                       1.0                     0.04762                 0.16667                 
CEN(Confusion entropy)                                           0                       0.52832                 0.5                     
DOR(Diagnostic odds ratio)                                       None                    1.25                    2.0                     
DP(Discriminant power)                                           None                    0.05343                 0.16597                 
DPI(Discriminant power interpretation)                           None                    Poor                    Poor                    
ERR(Error rate)                                                  0.0                     0.4                     0.4                     
F0.5(F0.5 score)                                                 1.0                     0.33333                 0.5                     
F1(F1 score - harmonic mean of precision and sensitivity)        1.0                     0.33333                 0.5                     
F2(F2 score)                                                     1.0                     0.33333                 0.5                     
FDR(False discovery rate)                                        0.0                     0.66667                 0.5                     
FN(False negative/miss/type 2 error)                             0                       2                       2                       
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     
FOR(False omission rate)                                         0.0                     0.28571                 0.33333                 
FP(False positive/type 1 error/false alarm)                      0                       2                       2                       
FPR(Fall-out or false positive rate)                             0.0                     0.28571                 0.33333                 
G(G-measure geometric mean of precision and sensitivity)         1.0                     0.33333                 0.5                     
IS(Information score)                                            1.73697                 0.152                   0.32193                 
J(Jaccard index)                                                 1.0                     0.2                     0.33333                 
MCC(Matthews correlation coefficient)                            1.0                     0.04762                 0.16667                 
MCEN(Modified confusion entropy)                                 0                       0.52877                 0.52832                 
MK(Markedness)                                                   1.0                     0.04762                 0.16667                 
N(Condition negative)                                            7                       7                       6                       
NLR(Negative likelihood ratio)                                   0.0                     0.93333                 0.75                    
NPV(Negative predictive value)                                   1.0                     0.71429                 0.66667                 
P(Condition positive or support)                                 3                       3                       4                       
PLR(Positive likelihood ratio)                                   None                    1.16667                 1.5                     
PLRI(Positive likelihood ratio interpretation)                   None                    Poor                    Poor                    
POP(Population)                                                  10                      10                      10                      
PPV(Precision or positive predictive value)                      1.0                     0.33333                 0.5                     
PRE(Prevalence)                                                  0.3                     0.3                     0.4                     
RACC(Random accuracy)                                            0.09                    0.09                    0.16                    
RACCU(Random accuracy unbiased)                                  0.09                    0.09                    0.16                    
TN(True negative/correct rejection)                              7                       5                       4                       
TNR(Specificity or true negative rate)                           1.0                     0.71429                 0.66667                 
TON(Test outcome negative)                                       7                       7                       6                       
TOP(Test outcome positive)                                       3                       3                       4                       
TP(True positive/hit)                                            3                       1                       2                       
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5                     
Y(Youden index)                                                  1.0                     0.04762                 0.16667                 
dInd(Distance index)                                             0.0                     0.72531                 0.60093                 
sInd(Similarity index)                                           1.0                     0.48713                 0.57508     
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.