# Finding optimal threshold in multi-class classification task

In a binary classification problem, it is easy to find the optimal threshold (F1) by setting different thresholds, evaluating them and picking the one with the highest F1. Similarly is there a proper way to find optimal thresholds for all the classes in a multi-class setting.

1. This will be a grid search problem if we do it brute force way. Any efficient way to do this?
2. Is there any package that ppl use or I can use for this?
3. Also is it common practice to find optimal threshold in multi-class settings, I couldn't any examples?
• Of possible interest: stats.stackexchange.com/questions/464636/…. Pay particular attention to the part of the answer that discusses having more decisions than categories! (I can see this happening if you have three categories and an output probability vector of $[0.33, 0.33, 0.34]$, which would tell me that the model has no idea about the category to which the observation belongs.)
– Dave
Dec 4, 2020 at 20:37

One solution is to explore a One-Vs-Rest classifier which creates separate binary classifiers for each class.

In a binary classification setting, you normally only have a single probability, and therefore you need a threshold to define the decision rule.

However, in multiclass classification problems where labels are mutually exclusive, you have a multinomial probability distribution, that is, the $$N$$ probabilities of the input belonging to each of the $$N$$ classes, all adding up to 1. In this kind of scenario, the decision rule usually is simply picking the highest probability class.

In the cases where labels are not mutually exclusive, each label's threshold can be individually selected just like in the binary classification case.

• What do you mean by "In the cases where labels are not mutually exclusive". Supervised Classification problem means that the labels are mutually exclusive.... am I missing good to learn content ? If you're correct, could you pls, provide a link to an example Jun 30, 2021 at 14:37
• There is a task called multi-label classification where there can be many labels associated, therefore not being mutually exclusive. Paperswithcode describes multiple benchmark datasets of such a type.
– noe
Jul 1, 2021 at 8:01
• Now I understand , thank you. I was aware of that multi-label classification, but it was unclear to me in the beginning what you meant. tnks Jul 1, 2021 at 15:29