2
$\begingroup$

Say I have three classes $C_1$,$C_2$, $C_3$ and a model $M$ which outputs a score $P$ for the confidence of each class for a sample $X$ i.e $M(X)=[P(C_1),P(C_2),P(C_3)]$ (note, we only want to predict one class)

Say I have created 3 one-vs-rest precision/recall plots and I decide that the optimal thresholds for each class are

$T_1 = 0.6$

$T_2 = 0.7$

$T_3 = 0.5$

We can then create the logic of assigning $X$ to a class like so:

If the index, $i$, of the biggest score of $M$ is greater than or equal to $T_i$, assign $X$ to $C_i$. Else, don't assign $X$ to anything, see the two examples below for two input of $X$:

$M(X_1) = [0.8,0.1,0.1] \rightarrow C_1\quad$ since the biggest socre is $0.8$ which is for class 1 and $T_1<0.8$

$M(X_2) = [0.3,0.6,0.1] \rightarrow \text{None}\quad$ since the biggest score is $0.6$ which is for class 2, but $T_2>0.6$

But, something tells me that in this way we don't preserve the optimal precision/recall for each class as we used to decide the thresholds at the first place, so my questions are:

  1. Can we have dynamic thresholds for a multi-class classification i.e a threshold for each class which preserves the optimal precision/recall for all classes at the same time?
  2. Is there a better way to decide thresholds for multi-class classification as per my problem above, when we want to control the precision/recall for each class

EDIT:

Say I have the following results for my validation-set:

conf  pred  target
----+------+-------
0.9    C1     C1
0.8    C1     C1
0.76   C1     C2
.
.
0.93   C2     C2
0.9    C2     C2
0.83   C2     C3
.
.

wouldn't this overcome the issue about the one-vs-rest example, since we now have the confidence when all three classes are involved and not 3x one-vs-rest?

$\endgroup$
3
  • $\begingroup$ I'm not clear how this confidence score is obtained? $\endgroup$
    – Erwan
    Apr 5 at 15:38
  • $\begingroup$ Say that it's pred_proba for any sklearn model/LightGBM (note I have omitted the confidences of the classes that did not have the highest confidence for the sake of simplicity) $\endgroup$
    – CutePoison
    Apr 5 at 17:51
  • $\begingroup$ Ok then I think it's still based on one-vs-rest classification, the score is relative to the other classes. It might be ok for your data/application, but in theory it should be interpreted carefully. As far as I know the multiclass classification doesn't offer a general way to adjust thresholds. $\endgroup$
    – Erwan
    Apr 6 at 10:56

1 Answer 1

0
$\begingroup$

I think there's a confusion between multi-class and multi-label classification:

  • In multi-class classification, every instance has a single label. This means that the classifier returns the single most likely label among the possible labels, not all the labels which may apply. In probabilistic terms, this implies that the output probabilities sum to 1 over all the classes.
  • In multi-label classification, every instance can have any number of labels (including no label at all). This is equivalent to training an independent model for each class. In probabilistic terms, every class probability $p$ represents the likelihood that the instance has this label as opposed to not having it.

Applying a custom threshold for each class in the multi-class case means that some instances may have zero or more than one class, i.e. it transforms the problem to multi-label.

So the problem should be clearly defined from the start:

  • If it's multi-class, then there's no way to use a custom threshold for every class, the most likely class is always the only one assigned.
  • If it's multi-label, then the classifiers are independent and custom thresholds can be used. But the problem is different, and in theory the training data should be consistent and contain instances with zero or multiple labels.

[edit following OP's comment]

I'm talking about multi-label because practically what you want to do is some kind of hybrid multi-class/multi-label classification. For example, in multi-class it's impossible for an instance not to have any label, as in your $X_2$ example, because there's always a most likely class ($C_2$ in your example).

It's important to understand that in one-vs-rest multi-class classification the predicted score cannot be interpreted independently, the classifier only knows how to distinguish between the possible classes. For example if one classifies images in three classes dog, fish and plant, an image of an elephant would be predicted as class dog with a very high probability because it's the closest. Whereas if one classifies dog vs. anything else, the probability of dog should be very low for the same image of elephant.

Using custom thresholds by class "breaks" the dependency that exists between the predicted scores for an instance, it's likely to cause a bias which advantages some class at the expense of the others.

Also I think that logically the possibility of zero label implies that multiple labels should also be possible, but I'm not sure that this is an important point.

$\endgroup$
4
  • $\begingroup$ It is indeed a multi-class classification problem (I can't see where I write it should be a multi-label problem. Agreed, that is two different things). The threshold is, per. the example, for deciding which of the three classes to return i.e we don't want to return C2 if the model output is less than 0.7 eventhough it's the highest score (see example 2) $\endgroup$
    – CutePoison
    Mar 23 at 12:03
  • 1
    $\begingroup$ I have updated my question to make it more clear. For your statement "If it's multi-class, then there's no way to use a custom threshold for every class, the most likely class is always the only one assigned." could you please explain why? Maybe if you have some papers that describe why $\endgroup$
    – CutePoison
    Mar 23 at 12:10
  • $\begingroup$ @CutePoison: Sorry maybe I didn't make it clear enough, see edit. $\endgroup$
    – Erwan
    Mar 23 at 12:47
  • $\begingroup$ THanks for the answer! I have edited my question $\endgroup$
    – CutePoison
    Apr 5 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.