Say I have a multiclass classification problem with
I have trained a classifier on a training set, I use a validation set and a One-vs-rest ROC-curve to give me
N ROC curves.
Since the ROC curve is created based on different thresholds of when we classify a sample as $Ci$ or not $Ci$. We can then chose (our) optimal FPR/TRP ratio and get the threshold (
t) e.g say
t=0.6 we classify a sample as $Ci$ if
model_score>=0.6 else "the rest" i.e not $Ci$.
(the blue marker at this picture from sklearn)
The question is, in the multi-class problem we can use e.g one-vs-rest and create
N ROC-curves (see below, also from sklearn)
but now we have
N different thresholds (in the plot,
N=3 since we have three classes). Say we in the one-vs-rest have defined the optimal threshold for the classes as
t1 = 0.8 (Class 1 vs rest) t2 = 0.6 (Class 2 vs rest) t3 = 0.4 (class 3 vs rest)
we get a new sample and the model-score is
S= [0.3,0.4,0.3] thus according to the thresholds we won't label it as any class since no score is above the threshold.