Multilabel image classification: is it necessary to have traning data for each combination of labels?

Question

I want to train a CNN for a multilabel image classification task using keras. However I am not sure how to prepare my tranining data. More specifically, I am wondering if I need training images that show a combination of two or more labels or if it is sufficient to train the network on single labels and it will then be able to detect multiple labels within an image.

I think my question becomes clearer with an example:

Say I am using the dog vs cat classification dataset and I want to build a model that is able to classify images as either being a dog or a cat or seeing both animals in one image. In this case, do I need to train the model with images showing cats, dogs AND images that show both in one image or is it sufficient to only have training images that only display cats and dogs?

Answer 1

Both cases will work. The point that you have to consider is that you shouldn't use Softmax activation in the last layer because your classes have intersection and they are not mutually exclusive. You should employ Sigmoid activation function and there should be two of them which each of them can be one or zero as the output vector. Consider that in this case you should not compare each activation to the other. Each one shows the probability of existence of the corresponding object, cat or dog.

For your case I suggest you to provide images which don't have either and in some cases have both. The former is more important. Because if you don't provide negative labels, not existing, the net always try to make something out of nothing. Means the net always try to flag similar patters as cats or dogs.

Answer 2

You need images from both labels, otherwise your CNN will predict any image as the label which you used for training. The model usually gets biased unless you use equal number of images for all labels. You can use data augmentation to generate more images so that all labels contain equal distribution.