How to split unbalanced data for supervised learning?

Question

Suppose I have data I want to use for supervised learning, but there is a pretty bad target/class/labels imbalance. Should I:

1. Limit the size of the training set to make sure there is a flat target/class balance distribution (the training set is designed such that there is an equal number of training samples for each class based on splitting the lowest-occuring class as high as possible). For example, if my lowest-occuring class appears only 50 times in my data, and I want an 80-20 train-test split, then I decide I take 40 of the samples for training, and for an even target balance, take 40 samples for all other samples in training, even if the highiest-occuring class appears 100,000 times, for instance.

2. Ignore target balance and just focus on the ratio for the train and testing split. So, if it's 80-20, take 40 of the samples out of 50 for my lowest-occuring class, and 80% of 100,000 for my highest occuring class, and so on.

3. Something else?

Let's suppose I can't just get more data. I know there's some stuff to be said regarding undersampling and oversampling, but what can I do to tell if either one is working better if model accuracy might be disingenuous?

Answer 1

It depends on what your goal is

• If every data point caries the same importance, then usually try to keep your validation/test set as close as possible to your later use case/true distribution. Maybe this https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html could help for data splitting. If you know that one class was over sampled than you can try some tweaking.
• If the rare classes are more important then you could change the distribution to their favor. Or penalize a mistake with a class weighting in the loss function.

After splitting off validation and test set you can try manipulating the training set or class weights in the loss function. Since then you can benchmark the performance of your changes to the desired validation/test set. Of course changing the data distribution or class weighting will introduce a bias and should be done carefully.

If you have really low class count you might want to resort to something more unorthodox than simple classification like outlier detection.

Answer 2

I would suggest three approaches:

• First use train_test_split(data, label, random_state= 2023, stratify= label) It will make sure that your training and validation/testing data will have similar imbalance.
• Second, choose the correct metric. Don't use accuracy in this case as it won't be a good metric.
• Third, you can use different techniques to make your data balance like SMOTE.