0
$\begingroup$

I'm working on Multi-Label problem i.e output can predict 1 or more label as an output and hence training data also have multiple labels.

Somehow I'm not able to map such ML model training. Please provide me the answer of below mentioned questions:

  1. How loss is getting calculated for such type of problem?
  2. How the ML model predicts?

I found one old thread Multi Label old query, but again Im not able to understand the solution.

$\endgroup$
1
$\begingroup$

In a normal multi-class setup we use softmax over last layer because we know that the sum of probability of the classes is 1 as only one of these class is the actual answer. After that we pass the output to a suitable loss function like BCE (binary cross entropy).

For multi-label we know that each class can be the output so the sum of probability of these classes will not necessarily be 1! However, the individual probability of each class still need to be between 0 and 1 to be a valid probability distribution. Therefore, simply apply a sigmoid layer on top of each class output and pass this to BCE loss which treats each class as a separate mini BCE loss and finally adds these losses across all the classes to get the final BCE.

For e.g., in Pytorch you just apply sigmoid and pass it to the BCE such that the dimensions of the output passed to BCE layer is (batch size, num of total class).

Look at the BCE loss section in the pytorch docs, where the dimensions of the input to BCE loss is (N, *) which implies the dimension which I just pointed out: https://pytorch.org/docs/stable/nn.html

$\endgroup$
  • $\begingroup$ For example we have 10 labels, does it mean we have to add 10 losses across all classes and treat them as a BCE individually? $\endgroup$ – vipin bansal Mar 4 '20 at 13:08
  • $\begingroup$ If you are implementing your own loss function then you are right that you need to create 10 BCE and add them to get the final BCE upon which you will backpropagate. However, if you are using a DL library like pytorch then they should be able to do that internally. $\endgroup$ – user1825567 Mar 4 '20 at 13:18
0
$\begingroup$

For multi class classification , after applying softmax over last layer we generally use categorical cross entropy as loss function rather than BCE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.