The customary objective function for multi-label (e.g. M labels) classification is binary cross-entropy. The problem is, if we use binary cross-entropy, we are assuming that the output labels are independent of each other, turning the problem to M independent binary classification problems. Is there any suitable objective function that makes the output labels to be dependent on each other?

  • $\begingroup$ I guess the main idea about the whole concept is that the random process are i.i.d. $\endgroup$ May 30, 2018 at 20:03
  • 1
    $\begingroup$ That objective would be an interesting objective to try, I am not aware of any! At the same time, how would you think one can take advantage of? What is your domain (target) that you are sure these labels are dependent? Take CelebA multilabel dataset (mmlab.ie.cuhk.edu.hk/projects/CelebA.html), some labels could be considered dependent Female/Lipstick/etc., but there are others are not, in which one could benefit from being them to be independent. $\endgroup$ Aug 12, 2018 at 18:31
  • 1
    $\begingroup$ @MajidMortazavi Thanks for time and consideration. The application is medical coding. The advantage is that different diseases have something in common. Thanks for introducing a new dataset to me. $\endgroup$
    – pythinker
    Aug 12, 2018 at 18:45
  • 1
    $\begingroup$ If it is a neural network then all the classification is done together till the last layer where they are separated by using binary cross entropy. My understanding is that in neural networks like for image recognition the recognition is connected on all previous layers except the last one. So for neural networks it is not a big deal that the last layer predicts separately every category. $\endgroup$
    – keiv.fly
    Oct 13, 2018 at 22:41

1 Answer 1


What you're looking for is called "cost-sensitive classification". Most methods however don't work with label similarities, but rather with relative penalties for different types of misclassifications.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.