Usage of Precision Recall on an unbalanced dataset

Question

I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.

Also the confusion matrix is of no help too.

I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall?

If anyone can shed any light on what approach should I take, that would really help me move forward.

Answer 1

When evaluating your algorithms, especially when your dataset is unbalanced, you should use more metrics than just accuracy. The accuracy is how many examples you have correctly identified in total. As you have seen if you have an unbalanced dataset where 0.5% of your instances are 1's then this will result in 99.5% accuracy if you blindly set all your outputs as zeroes. This is obviously wrong albeit the high accuracy. The accuracy is calculated as

$Accuracy = \frac{\sum{TP} + \sum{TN}}{\sum{TP} + \sum{TN} + \sum{FP} + \sum{FN}}$

where TP is true positive, TN is true negatives, FP is false positives and FN is false negatives.

If you want to capture the performance of your unbalanced dataset you should look into the percentage of FP and FN you are calculating. You can do this using the sensitivity and the specificity. Calculate the sensitivity as

$Sensitivity = \frac{\sum{TP} }{\sum{TP} + \sum{FN} }$

and the specificity as

$Specificity = \frac{\sum{TN} }{\sum{TN} + \sum{FP} }$.

An ideal classifier should have the accuracy, specificity and sensitivity all be 1. This would mean every sample is correctly classified. In your case where you are getting very high false negatives, you will see that your sensitivity will be very low. This is a measure with which you can state that your algorithm is performing poorly. It is good form to always include these metrics in any statistical study you are doing. Accuracy alone is not sufficient to prove that you are obtaining good results.

Moreover, there is the receiver-operator curve (ROC). This will tell you your false positive rate for any true positive rate. You can then calculate the area under this curve (AUC) to get a comparable metric of performance.

All of these should be used together when exclaiming the performance of your algorithm. The ROC and AUC can be omitted however leaving out the sensitivity and specificity of your algorithm is unwise.

Answer 2

You may use logistic regression if your dataset follows some clear polynomial curve, which you can verify by plotting the data and looking how it is distributed. In that case, even if the data is skewed you might get a good classifier.

The fact that you achieve higher accuracy - 99% as you mentioned - by always predicting one of the values is the reason why classifier accuracy is not used to evaluate a logistic regression model. Instead Precision and Recall give you much better insight into the quality of the classifier because they measure both how many of the examples it classified as positive were actually positive and how many of the positive examples in the training set it classified correctly.

The tradeoff between both metrics help you find a classifier which is both precise and can generalize.

Answer 3

Found a solution, for an unbalanced dataset, first use SMOTE and then apply any model to use check AUC