2
$\begingroup$

Say I have a multiclass classification problem with N classes. I have trained a classifier on a training set, I use a validation set and a One-vs-rest ROC-curve to give me N ROC curves.

Since the ROC curve is created based on different thresholds of when we classify a sample as $Ci$ or not $Ci$. We can then chose (our) optimal FPR/TRP ratio and get the threshold (t) e.g say t=0.6 we classify a sample as $Ci$ if model_score>=0.6 else "the rest" i.e not $Ci$. (the blue marker at this picture from sklearn) enter image description here

The question is, in the multi-class problem we can use e.g one-vs-rest and create N ROC-curves (see below, also from sklearn)

enter image description here

but now we have N different thresholds (in the plot, N=3 since we have three classes). Say we in the one-vs-rest have defined the optimal threshold for the classes as

t1 = 0.8 (Class 1 vs rest)
t2 = 0.6 (Class 2 vs rest)
t3 = 0.4 (class 3 vs rest)

we get a new sample and the model-score is S= [0.3,0.4,0.3] thus according to the thresholds we won't label it as any class since no score is above the threshold.

$\endgroup$
4
  • 1
    $\begingroup$ Please don't cross-post your question to multiple sites... stackoverflow.com/q/68405240/333599 $\endgroup$
    – Calimo
    Jul 16, 2021 at 8:32
  • 1
    $\begingroup$ See meta.stackexchange.com/q/64068/147320 $\endgroup$
    – Calimo
    Jul 16, 2021 at 8:33
  • $\begingroup$ What are the costs of making the various kinds of mistakes, such as calling a $0$ a $2$ or calling a $1$ a $0?$ $\endgroup$
    – Dave
    Jul 16, 2021 at 22:55
  • $\begingroup$ @Dave assume the misclassification-costs are all equal $\endgroup$
    – CutePoison
    Jul 18, 2021 at 6:48

1 Answer 1

1
$\begingroup$

Without modifying the assumptions or the model, there are two choices:

  1. The system returns the best single guess. For example, Class 2 since that is the best model's largest probability.
  2. The system returns "Not confident enough to make a prediction".
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.