10
$\begingroup$

I need to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue which is by up sampling or down sampling or by using Smote.

For example, if I have a rare disease 1 percent out of 100, and lets say I decided to have a balanced data set for my training set which is: 50/50 sample Will not that make the machine think 50% of patients will have disease? even though the ratio is 1 of 100. So

  1. Why do we need to deal with data imbalance?
  2. What is the recommended ratio to have balance set
$\endgroup$
  • $\begingroup$ I like that question. Understanding why is very important. $\endgroup$ – DaL Nov 6 '17 at 6:49
  • $\begingroup$ See a relate question in cross validated stats.stackexchange.com/questions/312780/… $\endgroup$ – DaL Nov 13 '17 at 7:31
7
$\begingroup$

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.

Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy, get 99% accuracy and go home early. The problem with such a model is that though it has high accuracy, it will probably not what you are looking for. Most of the time you want to find the people with the disease (giving high weigh to false negatives) much more than you are afraid to send an healthy person to unneeded test (low weight to false positives). In a real world health problem the ratio between the weight can easily be 1 to 1,000.

The imbalance in the distribution fails most algorithms from finding a proper solution.

You are correct that just balancing the distribution isn't the optimal solution. Indeed, an algorithm that is trained on a balanced distribution is not fitted to the natural distribution on which it will be evaluated. My favorite method is adapting it back, as you can see here. For a discussion, see here.

Just setting the ratio to some other problem won't work since you will have the same problem.

Smote is working in a different way, which didn't work as well when I tried it, but it might fit your problem.

$\endgroup$
  • $\begingroup$ So as I understand its better to balance the data because most algorithm works well on Balanced data even though it wont fit the natural distribution? $\endgroup$ – sara Nov 8 '17 at 10:33
  • $\begingroup$ Can you elaborate more on this sentence "when your cost of error doesn't fit the samples distribution." I didnt get it very well $\endgroup$ – sara Nov 8 '17 at 11:30
  • 1
    $\begingroup$ Answering the first comment: Imbalanced data set will lead algorithms to get good results by returning the majority. That will be a problem if you are interested in the minority more. So, balancing is a way to force the algorithm to give more weight to the minority. However, once you balanced, the train distribution and test distribution are different so you need to adapt your model back to the test distribution (like in the method I proposed). $\endgroup$ – DaL Nov 8 '17 at 11:48
  • $\begingroup$ Answering the second comment: Indeed, that point was a bit unclear. I edited the answer. Is it clear now? $\endgroup$ – DaL Nov 8 '17 at 11:49
  • $\begingroup$ yes so as I understand the training and test distribution should be the same, so wont it be a solution is that after I balance my training set to 50 50 ratio , I go and also set my test set to 50 50 ? $\endgroup$ – sara Nov 8 '17 at 12:21
3
$\begingroup$
  • Short answer:

you need to deal with class imbalance if/because it makes your model better (on unseen data). "Better" is something that you have to define yourself. It could be accuracy, it could be a cost, it could be the true positive rate etc.

  • Long answer:

There is a subtle nuance that is important to grasp when talking about class imbalance. Namely, is your data imbalanced because:

  1. the distribution of the data is itself imbalanced

In some cases, one class occurs much more than another. And it's okay. In this case, you have to look at whether certain mistakes are more costly than others. This is the typical example of detecting deadly diseases in patients, figuring out if someone is a terrorist etc. This goes back to the short answer. If some mistakes are more costly than others, you'll want to "punish" them by giving them a higher cost. Therefore, a better model will have a lower cost. If all mistakes are as bad, then there is no real reason why you should use cost sensitive models.

It's also important to note that using cost-sensitive models is not specific to imbalanced datasets. You can use such models if your data is perfectly balanced as well.

  1. it doesn't represent the true distribution of the data

Sometimes your data is "imbalanced" because it doesn't represent the true distribution of the data. In this case, you have to be careful, because you have "too many" examples of one class and "too few" of the other, and therefore, you need to make sure that your model doesn't over-/underfit on one of these classes.

This is different than using costs because it might not be the case that one mistake is worse than another. What would happen is that you would be biased and it wouldn't be beneficial for your model if the unseen data doesn't have the same distribution as the data you trained on.

Let's say that I give you training data and your goal is to guess if something is red or blue. Whether you mistake blue for red or red for blue doesn't make much of a difference. Your training data has 90% red instances where in real life, they only happen 10% of the time. You would need to deal with that in order to make your model better.

$\endgroup$
  • $\begingroup$ Very Elaborative answer. Can you explain more with example on what do u mean by "If your model is better because it has a low cost, then deal with the imbalance." Why i should deal with the imbalance if the model has a low cost? $\endgroup$ – sara Nov 9 '17 at 8:21
  • $\begingroup$ I edited my answer a little bit so hopefully, it's clearer. What I meant was the following: "if your model is considered better when it has a low cost, and dealing with the imbalance reduces the cost, then do it". $\endgroup$ – Valentin Calomme Nov 9 '17 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.