why we need to handle data imbalance?

Question

I need to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue which is by up sampling or down sampling or by using Smote.

For example, if I have a rare disease 1 percent out of 100, and lets say I decided to have a balanced data set for my training set which is: 50/50 sample Will not that make the machine think 50% of patients will have disease? even though the ratio is 1 of 100. So

1. Why do we need to deal with data imbalance?
2. What is the recommended ratio to have balance set

Answer 1

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.

Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy, get 99% accuracy and go home early. The problem with such a model is that though it has high accuracy, it will probably not what you are looking for. Most of the time you want to find the people with the disease (giving high weigh to false negatives) much more than you are afraid to send an healthy person to unneeded test (low weight to false positives). In a real world health problem the ratio between the weight can easily be 1 to 1,000.

The imbalance in the distribution fails most algorithms from finding a proper solution.

You are correct that just balancing the distribution isn't the optimal solution. Indeed, an algorithm that is trained on a balanced distribution is not fitted to the natural distribution on which it will be evaluated. My favorite method is adapting it back, as you can see here. For a discussion, see here.

Just setting the ratio to some other problem won't work since you will have the same problem.

Smote is working in a different way, which didn't work as well when I tried it, but it might fit your problem.

Answer 2

• Short answer:

you need to deal with class imbalance if/because it makes your model better (on unseen data). "Better" is something that you have to define yourself. It could be accuracy, it could be a cost, it could be the true positive rate etc.

• Long answer:

There is a subtle nuance that is important to grasp when talking about class imbalance. Namely, is your data imbalanced because:

1. the distribution of the data is itself imbalanced

In some cases, one class occurs much more than another. And it's okay. In this case, you have to look at whether certain mistakes are more costly than others. This is the typical example of detecting deadly diseases in patients, figuring out if someone is a terrorist etc. This goes back to the short answer. If some mistakes are more costly than others, you'll want to "punish" them by giving them a higher cost. Therefore, a better model will have a lower cost. If all mistakes are as bad, then there is no real reason why you should use cost sensitive models.

It's also important to note that using cost-sensitive models is not specific to imbalanced datasets. You can use such models if your data is perfectly balanced as well.

2. it doesn't represent the true distribution of the data

Sometimes your data is "imbalanced" because it doesn't represent the true distribution of the data. In this case, you have to be careful, because you have "too many" examples of one class and "too few" of the other, and therefore, you need to make sure that your model doesn't over-/underfit on one of these classes.

This is different than using costs because it might not be the case that one mistake is worse than another. What would happen is that you would be biased and it wouldn't be beneficial for your model if the unseen data doesn't have the same distribution as the data you trained on.

Let's say that I give you training data and your goal is to guess if something is red or blue. Whether you mistake blue for red or red for blue doesn't make much of a difference. Your training data has 90% red instances where in real life, they only happen 10% of the time. You would need to deal with that in order to make your model better.