Identify optimal thresholds for one-vs-one/one-vs-rest ROC-curve for multiclass classification

Say I have a multiclass classification problem with N classes. I have trained a classifier on a training set, I use a validation set and a One-vs-rest ROC-curve to give me N ROC curves.

Since the ROC curve is created based on different thresholds of when we classify a sample as $$Ci$$ or not $$Ci$$. We can then chose (our) optimal FPR/TRP ratio and get the threshold (t) e.g say t=0.6 we classify a sample as $$Ci$$ if model_score>=0.6 else "the rest" i.e not $$Ci$$. (the blue marker at this picture from sklearn)

The question is, in the multi-class problem we can use e.g one-vs-rest and create N ROC-curves (see below, also from sklearn)

but now we have N different thresholds (in the plot, N=3 since we have three classes). Say we in the one-vs-rest have defined the optimal threshold for the classes as

t1 = 0.8 (Class 1 vs rest)
t2 = 0.6 (Class 2 vs rest)
t3 = 0.4 (class 3 vs rest)


we get a new sample and the model-score is S= [0.3,0.4,0.3] thus according to the thresholds we won't label it as any class since no score is above the threshold.

• Please don't cross-post your question to multiple sites... stackoverflow.com/q/68405240/333599 Jul 16, 2021 at 8:32
• Jul 16, 2021 at 8:33
• What are the costs of making the various kinds of mistakes, such as calling a $0$ a $2$ or calling a $1$ a $0?$
– Dave
Jul 16, 2021 at 22:55
• @Dave assume the misclassification-costs are all equal Jul 18, 2021 at 6:48