What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?

For example I have apple and pear pictures. What I am trying to do is to predict if a picture is an apple or pear picture and AT THE SAME TIME predicting whether the fruit is big and/or yellow.

Thus predicting the fruit type (apple/pear) is multi-class problem, one vs all. Predicting whether the fruit big and/or yellow is multi-label problem.

If label ordering is [apple, pear, big, yellow] then a “big yellow apple” picture should have the label [1,0,1,1]. First two part is mutually exclusive, one-hot, however last two are not.

So, what loss function should I use for this problem?

• You can use a compound loss function that adds the multi-label and binary classification losses. The trick is setting the relative importance of one against the other. Welcome to the site! – Emre Jan 19 '18 at 18:55

My opinion is that for this problem you don't have to make a new cost function. For the mutually exclusive part, as you have truly stated that, and for the second part of your vector, the well known cross-entropy cost function will perform fine. I guess the problem is something else. For problems with mutually exclusive classes, we use soft max layer as the last layer for neural nets while for cases which classes are not mutually exclusive, you can use sigmoid as the activation function. In your case that you have combination of them I suggest you an alternative approach:
Change your mutually exclusive part to a binary output, means if the corresponding entry is less than half, you can understand that it is e.g. an apple otherwise, it is the other class. and for the rest, just keep the output vector as it is. Finally use sigmoid activation function as the last layer if you are using neural nets.