2
$\begingroup$

I have been trying to evaluate my models used on fire systems dataset with a huge imbalance in the dataset. Most models failed to predict any true positives correctly however naive Bayes managed to do that but with a very high rate of False Positive. I had run the experiments on both the confusion matrix and classification report for both can be seen below. The same dataset and train/test split was used with both of the datasets

 Naive Bayes Confusion Matrix and Classification Report

     [[TN=732 FP=448]
     [FN=2   TP=15]]


          precision    recall  f1-score   support

       0       1.00      0.62      0.76      1180
       1       0.03      0.88      0.06        17

accuracy                               0.62      1197
macro avg          0.51      0.75      0.41      1197
weighted avg       0.98      0.62      0.75      1197


Logistic Regression Confusion Matrix and Classification Report


     [[TN=1180 FP=0]
     [FN=17   TP=0]]


          precision    recall  f1-score   support

       0       0.99      1.00      0.99      1180
       1       0.00      0.00      0.00        17

accuracy                              0.98      1197
macro avg          0.49      0.50     0.50      1197
weighted avg       0.97      0.99     0.98      1197

However I got the Kohen Kappa Coefficient for these models and I am quite confused on how to interpret the values. Please find values below

Logistic Regression=0.0
Naive Bayes=0.03

These values indicate very slight agreement. But why is the value of Naive Bayes slightly better than Logistic regression ?

Improve this question
$\endgroup$
1
  • $\begingroup$ Do you mean Cohen and Kohen one and same person? $\endgroup$
    – Subhash C. Davar
    Commented Mar 23, 2020 at 13:11

1 Answer 1

Reset to default
2
$\begingroup$

Logistic Regression is only predicting one class (in this case the negative class)! Because of the high imbalance in the data, this model gives a high accuracy score. This metric, however, isn't reliable for imbalanced datasets. A more proper metric like Cohen's Kappa penalizes this behavior.

Naive Bayes, on the other hand, tries to predict both classes. It misses a lot more predictions this way, but it's Kappa is higher.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What is imbalanced dataset? $\endgroup$
    – Subhash C. Davar
    Commented Mar 23, 2020 at 1:02
  • $\begingroup$ Imbalance is if one class has more samples than the other(s). In your case you have 1180 samples in class 0 and 17 samples in class 1. This means that by predicting just class 0 (i.e. a completely dumb classifier), you'll get an accuracy of 1180/1197=98.58%. This seems like a good number but it's not. Accuracy isn't a good metric in this case for this exact reason. $\endgroup$
    – Djib2011
    Commented Mar 23, 2020 at 8:59
  • $\begingroup$ Cohen's Kappa penalizes this behavior. Why Kohen in question ? $\endgroup$
    – Subhash C. Davar
    Commented Mar 23, 2020 at 9:07
  • 1
    $\begingroup$ It's a bit more complicated metric. It takes into account both classes for computing its value. Look at this for an example. What it boils down to is that it's a more reliable metric for imbalanced datasets. $\endgroup$
    – Djib2011
    Commented Mar 23, 2020 at 9:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.