The effect of imbalanced distribution of data

Question

I read on Google's ML website if I have classification dataset with a ratio of 90% for one classification and 10% of the data for another classification.

In that case, should I use the exact same percentage of data for each classification? i.e. deleting around 80% of the dataset to make it 10% for each classification.

The reason is that Google said that the ML model will learn and then it is more likely to have a classification of the 90% and that won't provide good predictions. (i.e) The predictions might be biased towards a single label/feature.

My dataset is 90% to 10% but that is indeed the actual ratio and it is more likely to have the classification of the 90%

Shall I delete 80% of my data or keep it as is and let the ML learn that it is indeed more likely to have a classification of the 90%?

Answer 1

The best way forward here depends highly on the real life question you try to answer.

Let's say you want to make a medical diagnosis:

'Sick with exotic Illness X' or 'Not sick with exotic illness X' in this case you might want to catch all instances of being sick as a warning sign and could live with 'false positives'.

Conversely your algorithm will be used to predict 'customers likely to cancel soon', in this case it would not be a good idea to proactively talk to 'false positives' i.e. customers who did not plan to cancel about why they might be dissatisfied.

In either cases your training set and indeed reality might be severely unbalanced but the cost and consequences of this varies.

In the first case I would recommend using balancing methods (like the aforementioned Under-/Oversampling, etc.) to improve recognition of the minority class while in the second case that might be unnecessary.

In any case I would practically go on to do the following:

Include balancing/sampling in your beauty contest of algorithms and parameters and check the impact on the accuracy of predicting the test set (which is left unbalanced as in the original).

This will simply show you whether the inherent bias of the training set is problematic for your real world case (i.e. produces models that never identify the minority class) or not.

Answer 2

You can also have a look at the concept of class weight. For example, you can assign a weight of 9 for minority class. It means, that each point in the minority class is 9 times more important, than each point in the majority class.

If you are using python and scikit-learn, for many classifieres you can set

class_weight='balanced'

Answer 3

Methods like Undersampling and Oversampling is the basic intuition you should start looking for when you encounter a class imbalance in your dataset. It has been already answered here