4
$\begingroup$

I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.

Also the confusion matrix is of no help too.

I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall?

If anyone can shed any light on what approach should I take, that would really help me move forward.

$\endgroup$
  • 1
    $\begingroup$ You should weight the business value/cost of your false positives and false negatives and determine if you would prefer higher precision or higher recall. $\endgroup$ – jeweinb Jul 25 '17 at 19:14
4
$\begingroup$

When evaluating your algorithms, especially when your dataset is unbalanced, you should use more metrics than just accuracy. The accuracy is how many examples you have correctly identified in total. As you have seen if you have an unbalanced dataset where 0.5% of your instances are 1's then this will result in 99.5% accuracy if you blindly set all your outputs as zeroes. This is obviously wrong albeit the high accuracy. The accuracy is calculated as

$Accuracy = \frac{\sum{TP} + \sum{TN}}{\sum{TP} + \sum{TN} + \sum{FP} + \sum{FN}}$

where TP is true positive, TN is true negatives, FP is false positives and FN is false negatives.

If you want to capture the performance of your unbalanced dataset you should look into the percentage of FP and FN you are calculating. You can do this using the sensitivity and the specificity. Calculate the sensitivity as

$Sensitivity = \frac{\sum{TP} }{\sum{TP} + \sum{FN} }$

and the specificity as

$Specificity = \frac{\sum{TN} }{\sum{TN} + \sum{FP} }$.

An ideal classifier should have the accuracy, specificity and sensitivity all be 1. This would mean every sample is correctly classified. In your case where you are getting very high false negatives, you will see that your sensitivity will be very low. This is a measure with which you can state that your algorithm is performing poorly. It is good form to always include these metrics in any statistical study you are doing. Accuracy alone is not sufficient to prove that you are obtaining good results.

Moreover, there is the receiver-operator curve (ROC). This will tell you your false positive rate for any true positive rate. You can then calculate the area under this curve (AUC) to get a comparable metric of performance.

All of these should be used together when exclaiming the performance of your algorithm. The ROC and AUC can be omitted however leaving out the sensitivity and specificity of your algorithm is unwise.

| improve this answer | |
$\endgroup$
  • $\begingroup$ While I agree with this, you haven't addressed the OP's interest in precision and recall. (Also, while it doesn't really matter either way, the reverted edit was correct: the term is "receiver operating characteristic curve," abbreviated to "ROC curve.") $\endgroup$ – Ben Reiniger Mar 29 at 19:29
2
$\begingroup$

You may use logistic regression if your dataset follows some clear polynomial curve, which you can verify by plotting the data and looking how it is distributed. In that case, even if the data is skewed you might get a good classifier.

The fact that you achieve higher accuracy - 99% as you mentioned - by always predicting one of the values is the reason why classifier accuracy is not used to evaluate a logistic regression model. Instead Precision and Recall give you much better insight into the quality of the classifier because they measure both how many of the examples it classified as positive were actually positive and how many of the positive examples in the training set it classified correctly.

The tradeoff between both metrics help you find a classifier which is both precise and can generalize.

| improve this answer | |
$\endgroup$
-1
$\begingroup$

Found a solution, for an unbalanced dataset, first use SMOTE and then apply any model to use check AUC

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.