2
$\begingroup$

I like to understand what is the accuracy of an imbalanced dataset.

Let's suppose we have a medical dataset and we want to predict the disease among the patients. Say, in an existing dataset 95% of patients do not have a disease, and 5% patients have disease. So clearly, it is an imbalanced dataset. Now, assume our model predicts that all 100 out of 100 patients have no disease.

Accuracy means = (TP+TN)/(TP+TN+FP+FN)

If the model predicts 100 patients do not have a disease and we are predicting disease among the patient then True positive refers to the disease among the patient and True negative refers to no disease among the patient.

In that case accuracy should be (0+100)/(0+100+0+0) = 1.

We are going to predict how many patients have a disease so if we get accuracy 1, does that mean 100% of patients have the disease?

I am taking the example from 5 Techniques to Handle Imbalanced Data For a Classification Problem . I am not sure at the time of accuracy calculation why they calculate it as (0+95)/(0+95+0+5) = 0.95, if they have already described that their model predicts all 100 out of 100 patients have no disease.

I hope I clarified my question. Thank you.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

Accuracy is the number of correct predictions out of the number of possible predictions. In many regards, it is like an exam score: you had an opportunity to get $100\%$ of the points and got $97\%$ or $79\%$ or whatever. The class ratio is not a factor.

In your example, you had $95$ negative patients and $5$ positive. You predicted $100$ negative patients, meaning that you got $95$ correct and $5$ incorrect for an accuracy of $95\%$.

Note that accuracy is a surprisingly problematic measure of performance, and this is true even when the classes are naturally balanced.

With imbalance, however, accuracy has the potential to mislead in a way that is not present in many other measures of performance, and your example is a good demonstration of that. All your model does is predict the majority class; it does nothing clever. However, your model achieves an accuracy of $95\%$, which sounds like a high $\text{A}$ in school that indicates strong performance.

Improve this answer
$\endgroup$
3
  • $\begingroup$ If you take an approach like $R^2$ in regression and compare to a baseline model, you will find your $R^2$-style measure of performance to be zero: exactly the same as the baseline performance that requires no fancy modeling by highly paid data scientists. $\endgroup$
    – Dave
    Commented Sep 10, 2022 at 18:24
  • $\begingroup$ Thank you for the explanation. So, the accuracy it got for the accuracy of patients who have diseases. However, at the time of training, it did not recognize any instances as True positives. So, in this respect, the accuracy is misleading here. Right? $\endgroup$
    – Encipher
    Commented Sep 10, 2022 at 18:30
  • 2
    $\begingroup$ Accuracy only ever says exactly what it claims to say, so there is a sense in which it cannot be misleading. Humans probably do not tend to think this way, however, and I would expect most to be expect more from accuracy than just how many correct predictions out of total predictions. In other words, the English word “accuracy” probably has implications that are not a part of the formal definition of “accuracy” in machine learning. $\endgroup$
    – Dave
    Commented Sep 10, 2022 at 18:50
5
$\begingroup$

Your calculation of the accuracy is incorrect, since the value for TN should not be 100 but 95 as in the example. The model predicts 100 patients to have no disease, but is actually only correct for 95 of those 100 patients since 5 of them actually do have the disease. Therefore the true negative rate is actually 95 instead of 100.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So, True negative = 95, patients do not have cancer and then the 5 patients who also recognize as true negative they are actually false negative. Therefore the above accuracy calculation should be (TP+TN)/ (TP+TN+FP+FN) means (0+95)/(0+95+0+5). Am I correct? $\endgroup$
    – Encipher
    Commented Sep 10, 2022 at 18:25
  • 1
    $\begingroup$ This is correct, the only change I would make is remove the word true from "the 5 patients who also recognize as true negative they are actually false negative" so it says "the 5 patients who are also recognized as negative are actually false negative". $\endgroup$
    – Oxbowerce
    Commented Sep 10, 2022 at 19:00
  • 1
    $\begingroup$ Yeah you are right they are predicted as negative but they are not true negative only the predicted negative. I should be careful with my words. Thank you. $\endgroup$
    – Encipher
    Commented Sep 10, 2022 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.