3
$\begingroup$

What precautions do I need to take while trying to develop a CNN for classification of images if there is much more training data for one label. For example:

label1 : 1000 images
label2 : 100 images
label3 : 100 images
label4 : 100 images

Numbers will become larger later but proportion is likely to stay the same.

Thanks for your insight.

$\endgroup$
4
$\begingroup$

The problem you face is commonly called the class imbalance and has been the subject of quite a bit of research. Here's a literature review, if you're interested: He, H., & Garcia, E. A. (2008). Learning from imbalanced data

In particular, you might encounter two forms of imbalance:

Absolute imbalance/rarity occurs when, while you have plenty of data from some classes, you have only a few examples of some other classes (or subconcept of a class). In this case, the issue is that there may not be enough data for the learning algorithm to learn the minority class. In the example you give, with 100 examples of the minority class, depending on the nature of your data, you might have this problem. If you expect to have more data in the future, however, absolute rarity should eventually no longer be an issue.

Relative imbalance, on the other hand, does not go away with more data. You have a relative imbalance when the prior probability of certain classes is much larger than that of some other class or classes. For example, you will always have 10 times as many examples of class 1 than examples of class 2, because class 1 occurs 10 times as often.

Most learning algorithms for classification optimize for accuracy, or something similar like RMSE. This means that, when solving the real classification problem is hard enough, and the data is strongly imbalanced towards one class, the model may resort to predicting the majority class whenever there's a doubt. Recall for the majority class may be great, but not so much for the minority class.

This becomes an issue in many domains where detection of the minority class is particularly important. For example, in medical diagnoses, we may be willing to sacrifice overall accuracy (because of more false positives) in order to have a better true positive rate.

In short, it depends on your domain. Are you OK with optimizing for overall accuracy, or is it more important to have comparable performances across the classes? If you choose the latter, then there are a few things you may try:

  • Use cost-sensitive learning: Certain learning algorithms and implementations allow you to assign a cost to each class, essentially describing how bad it is if an example of that class is misclassified. If I recall correctly, this is usually considered the best approach when you have a good idea of these different costs.
  • Rebalance the classes: You can oversample the minority class (which carries a risk of overfitting), undersample the majority class (which is dangerous if you don't have a lot of data), use a mixture of both or perhaps something a bit more advanced like synthetic sampling (attempt to generate new examples of the minority class using something like SMOTE)

Overall, you should also be careful to pick the right evaluation metric. Evaluating your model using accuracy may lead you to believe that your model is performing very well when in fact it is classifying everything into the majority class. There are many metrics you may use, each of which have their pros and cons. The area under the ROC curve (AUC) is a common metric which gives you a general idea of your model's performance averaged over different class misclassification costs. If you can plot the ROC curve for multiple models and notice that one curve dominates the others across the entire width of the plot, then that is the clearest sign that you have a winner. There's a whole chapter on the subject, if you're interested, in the book Imbalanced Learning: Foundations, Algorithms, and Applications.

$\endgroup$
  • $\begingroup$ Thanks for a great explanation. And welcome to stackexchange! Why have you not been contributing here earlier? $\endgroup$ – rnso Sep 26 '18 at 14:02
  • $\begingroup$ @mso, if you are appreciative of the answer, would be great if you upvoted it. See here: datascience.meta.stackexchange.com/a/2340/29575 $\endgroup$ – Stephen Rauch Sep 26 '18 at 14:35
  • $\begingroup$ I actually accepted it promptly. An upvote has also been delivered. $\endgroup$ – rnso Sep 26 '18 at 15:23
  • $\begingroup$ Glad you appreciate it :) $\endgroup$ – Vincent B. Lortie Sep 26 '18 at 15:26
2
$\begingroup$

You can duplicate the images and add them. You can use data augmentation techniques for the labels which have less images. The below code is for Keras.

datagen = ImageDataGenerator(
  rotation_range=40,
  width_shift_range=0.2,
  height_shift_range=0.2,
  shear_range=0.2,
  zoom_range=0.2,
  horizontal_flip=True,
  fill_mode='nearest')

I hope this helps. You should not be worried about one label having more data, rather you should think how to increase data for other labels.

$\endgroup$
  • $\begingroup$ Is vertical_flip also useful? Can it be done with this function? Also, what do you think of augmentation technique given in this post: datascience.stackexchange.com/questions/38795/… ? $\endgroup$ – rnso Sep 26 '18 at 8:30
  • $\begingroup$ I think it will work if you change the (8 x 8) according to your image size but it's always better to define a new function to suit your needs. $\endgroup$ – Danny Sep 26 '18 at 8:40
  • $\begingroup$ You should first train your model on the unbalanced training set and check your results. These may serve as a baseline for further optimization. You can try different settings as well, like making sure that your batches have at least one example of each class. What I mean is: first check whether the unbalanced classes are in fact a problem, before trying to solve it $\endgroup$ – qmeeus Sep 26 '18 at 8:46
2
$\begingroup$

The dataset you are using contains almost above 90% of the training data belonging to one single class and will greatly impact your results. This imbalance of the data is ought to generate what we call as Skew classes. Presence of skew classes is going to influence your predictions and the learned model could become one that predicts the majority class.

In order to overcome this problem, you can do the following:

  • Sampling: Up sample or down sample your dataset to ensure equal representations of the data.
  • Discarding excess data: If the data in other classes is sufficient, simply discard some data from the dominating class.
  • Weighting: Certain training algorithms take weights to put emphasis on the classes and could be helpful during skew classes.

This answer is based on this article. Refer it for detailed explaination.

$\endgroup$
  • $\begingroup$ this answer seems promising, but have a look at this post in order to improve your answer. Look especially at the Provide context for links section. You can summarize the main points of the articles you shared in case the links are removed. $\endgroup$ – BrunoGL Sep 26 '18 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.