What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?

Question

For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.

Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.

If label ordering is [apple, pear, big, yellow] then a “big yellow apple” picture should have the label [1,0,1,1]. First two part is mutually exclusive, one-hot, however last two are not.

So, what loss function should I use for this problem?

Answer 1

The labels of data are not mutually exclusive so you can't say this is a one vs. all problem, because more than one entry may be one in the output vector. Moreover, if in the seen there should be an apple or pear this can be considered as an exhaustive problem which means one of them should happen for each input.

My opinion is that for this problem you don't have to make a new cost function. For the mutually exclusive part, as you have truly stated that, and for the second part of your vector, the well known cross-entropy cost function will perform fine. I guess the problem is something else. For problems with mutually exclusive classes, we use soft max layer as the last layer for neural nets while for cases which classes are not mutually exclusive, you can use sigmoid as the activation function. In your case that you have combination of them I suggest you an alternative approach:

Change your mutually exclusive part to a binary output, means if the corresponding entry is less than half, you can understand that it is e.g. an apple otherwise, it is the other class. and for the rest, just keep the output vector as it is. Finally use sigmoid activation function as the last layer if you are using neural nets.