Suppose I have data I want to use for supervised learning, but there is a pretty bad target/class/labels imbalance. Should I:
Limit the size of the training set to make sure there is a flat target/class balance distribution (the training set is designed such that there is an equal number of training samples for each class based on splitting the lowest-occuring class as high as possible). For example, if my lowest-occuring class appears only 50 times in my data, and I want an 80-20 train-test split, then I decide I take 40 of the samples for training, and for an even target balance, take 40 samples for all other samples in training, even if the highiest-occuring class appears 100,000 times, for instance.
Ignore target balance and just focus on the ratio for the train and testing split. So, if it's 80-20, take 40 of the samples out of 50 for my lowest-occuring class, and 80% of 100,000 for my highest occuring class, and so on.
Something else?
Let's suppose I can't just get more data. I know there's some stuff to be said regarding undersampling and oversampling, but what can I do to tell if either one is working better if model accuracy might be disingenuous?