8
$\begingroup$

I am trying to solve a multiclass classification problem. The dataset is balanced. I have been using accuracy as a performace metric till now. Are there any other good performance metrics for this task?

I already know about precision and recall but as far as I know they are used when the dataset is imbalanced.

$\endgroup$
  • $\begingroup$ you mean the target variable is balanced in both train and test right? it would be great if you can explain a bit on your business as well. $\endgroup$ – Toros91 May 7 '18 at 5:48
  • $\begingroup$ There is no "good performance metric". It really depends on what your task is. If you understand your model and output with accuracy only, then it is a good metric for you. $\endgroup$ – Ankit Seth May 7 '18 at 7:06
  • $\begingroup$ Have you looked at kappa? $\endgroup$ – HEITZ May 8 '18 at 22:09
2
$\begingroup$

If you can use python I suggest PyCM module. A vast variety of performance evaluation parameters is in access by this module and also you can use its documentation if you want to implement it by yourself.

There is an example of it:

>>> from pycm import *
>>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
>>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])
>>> cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred) # Create CM From Data
>>> cm.classes
[0, 1, 2]
>>> cm.table
{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}
>>> print(cm)
Predict          0        1        2        
Actual
0                3        0        0        
1                0        1        2        
2                2        1        3        




Overall Statistics : 

95% CI                                                           (0.30439,0.86228)
Bennett_S                                                        0.375
Chi-Squared                                                      6.6
Chi-Squared DF                                                   4
Conditional Entropy                                              0.95915
Cramer_V                                                         0.5244
Cross Entropy                                                    1.59352
Gwet_AC1                                                         0.38931
Joint Entropy                                                    2.45915
KL Divergence                                                    0.09352
Kappa                                                            0.35484
Kappa 95% CI                                                     (-0.07708,0.78675)
Kappa No Prevalence                                              0.16667
Kappa Standard Error                                             0.22036
Kappa Unbiased                                                   0.34426
Lambda A                                                         0.16667
Lambda B                                                         0.42857
Mutual Information                                               0.52421
Overall_ACC                                                      0.58333
Overall_RACC                                                     0.35417
Overall_RACCU                                                    0.36458
PPV_Macro                                                        0.56667
PPV_Micro                                                        0.58333
Phi-Squared                                                      0.55
Reference Entropy                                                1.5
Response Entropy                                                 1.48336
Scott_PI                                                         0.34426
Standard Error                                                   0.14232
Strength_Of_Agreement(Altman)                                    Fair
Strength_Of_Agreement(Cicchetti)                                 Poor
Strength_Of_Agreement(Fleiss)                                    Poor
Strength_Of_Agreement(Landis and Koch)                           Fair
TPR_Macro                                                        0.61111
TPR_Micro                                                        0.58333

Class Statistics :

Classes                                                          0                       1                       2                       
ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 
BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667                 
DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     
ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 
F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692                 
F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 
F2(F2 score)                                                     0.88235                 0.35714                 0.51724                 
FDR(False discovery rate)                                        0.4                     0.5                     0.4                     
FN(False negative/miss/type 2 error)                             0                       2                       3                       
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     
FOR(False omission rate)                                         0.0                     0.2                     0.42857                 
FP(False positive/type 1 error/false alarm)                      2                       1                       2                       
FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333                 
G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 
LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5                     
LR-(Negative likelihood ratio)                                   0.0                     0.75                    0.75                    
MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 
MK(Markedness)                                                   0.6                     0.3                     0.17143                 
N(Condition negative)                                            9                       9                       6                       
NPV(Negative predictive value)                                   1.0                     0.8                     0.57143                 
P(Condition positive)                                            3                       3                       6                       
POP(Population)                                                  12                      12                      12                      
PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6                     
PRE(Prevalence)                                                  0.25                    0.25                    0.5                     
RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 
RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007                 
TN(True negative/correct rejection)                              7                       8                       4                       
TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667                 
TON(Test outcome negative)                                       7                       10                      7                       
TOP(Test outcome positive)                                       5                       2                       5                       
TP(True positive/hit)                                            3                       1                       3                       
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5  

>>> cm.matrix()
Predict          0        1        2        
Actual
0                3        0        0        
1                0        1        2        
2                2        1        3        

>>> cm.normalized_matrix()
Predict          0              1              2              
Actual
0                1.0            0.0            0.0            
1                0.0            0.33333        0.66667        
2                0.33333        0.16667        0.5 
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Please also explain which metrics you think are the most valuable and why. $\endgroup$ – Vlad Jan 21 '19 at 10:03
  • 1
    $\begingroup$ It depends on your case and the problem in which you are trying to classify the data but at the most cases, kappa and the Strength_Of_Agreement(Landis and Koch) are more reliable. $\endgroup$ – Alireza Zolanvari Jan 21 '19 at 17:40
  • $\begingroup$ In new version of PyCM (v 1.9) a recommender system had been added. In your case (balanced dataset and multi-class classification) the following parameters is suggested: ERR, TPR Micro, TPR Macro, PPV Micro, PPV Macro, ACC, Overall ACC, MCC, Overall MCC, BCD, Hamming Loss, Zero-one Loss for more information about this parameters visit the here $\endgroup$ – Alireza Zolanvari Mar 2 '19 at 10:33
1
$\begingroup$

Take a look at multi-class confusion matrix. maybe the model has some difficulty on a subset of the classes.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Multi class log-loss is used in many data science competitions. Although not so easily interpretable as accuracy, it penalizes based on your confidence on your predictions. If the models you are using output probabilities, it may be a better way than accuracy to compare and select different models on your validation data, as it takes into account the probabilities and not only the amount of correct predictions.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Multi-Class log-loss is a proxy for the zero-one loss because the latter is not differentiable if we are doing back-prob. in neural networks regime we could call it as simply softmax function. but the thing is we want to define a metric to measure the performance. sometimes we use hyper-parameter search to find the optimal threshold that could optimize our main subjective metric. e.g. we want to put more penalty on some class. $\endgroup$ – Fadi Bakoura May 7 '18 at 7:42
  • 3
    $\begingroup$ Multi-class log-loss is not exclusively defined for neural networks. It is a metric on how well a classifier is doing itself and by definition it does not necessarily have anything to do with backpropagation. $\endgroup$ – David Masip May 7 '18 at 7:43
  • $\begingroup$ yeah, you are right, maybe we want to know what are the examples that are indistinguishable, we could simply check the actual loss to measure the distance between the two distributions (data/model). yet this is a quantitative analysis, my suggestion was to start with some kind of qualitative analysis. $\endgroup$ – Fadi Bakoura May 7 '18 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.