Creating dataset - imbalanced or balanced?

Question

I'm trying to make an image classification model and I have 5 classes - A, B, C, D, E. The goal is to get the highest possible classification accuracy.

I have a database of images and I'm selecting the number of images I will use for each class for my model. I'm trying to figure out how many images I should pick for each class if the distribution of available data is as shown below.

Should I randomly choose something like 8,000 images for each class, in order to avoid class imbalance? Or should I just use as many images per class?

Images available per class:

• A - 100,000
• B - 70,000
• C - 40,000
• D - 10,000
• E - 8,000

Answer 1

You'll first have to verify if this class imbalance is really an issue. Most of the datasets available are imbalanced in their distribution. But they may or may not affect the performance of the classifiers. For some algorithms, the number of samples in the minority-class (class E in this case) is sufficient to train a reliable classifier. So, it can be best understood by looking a the confusion matrix to see if the model is biased to the majority class. If the model is not really biased by the majority class, it's good, otherwise you may have to think about a strategy to handle class imbalance problem.

You can also try to balance the classes by removing the samples from majority class (make all the class to have same number of samples, 8000 in your case), and check the extent to which the confusion matrix has changed. This shows if the class imbalance problem exists or not.

Answer 2

The easiest option to close your approach is too just run the model twice in both variations. But in my experience this specific data is bound to be imbalanced.

Your idea to fix the class imbalance is technically known as under-sampling. This is one technique which is used. You can try out a few other techniques which are mentioned here: https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html