3
$\begingroup$

In a binary classification problem, it is easy to find the optimal threshold (F1) by setting different thresholds, evaluating them and picking the one with the highest F1. Similarly is there a proper way to find optimal thresholds for all the classes in a multi-class setting.

  1. This will be a grid search problem if we do it brute force way. Any efficient way to do this?
  2. Is there any package that ppl use or I can use for this?
  3. Also is it common practice to find optimal threshold in multi-class settings, I couldn't any examples?
$\endgroup$
1
  • $\begingroup$ Of possible interest: stats.stackexchange.com/questions/464636/…. Pay particular attention to the part of the answer that discusses having more decisions than categories! (I can see this happening if you have three categories and an output probability vector of $[0.33, 0.33, 0.34]$, which would tell me that the model has no idea about the category to which the observation belongs.) $\endgroup$
    – Dave
    Dec 4 '20 at 20:37
0
$\begingroup$

One solution is to explore a One-Vs-Rest classifier which creates separate binary classifiers for each class.

$\endgroup$
0
$\begingroup$

In a binary classification setting, you normally only have a single probability, and therefore you need a threshold to define the decision rule.

However, in multiclass classification problems where labels are mutually exclusive, you have a multinomial probability distribution, that is, the $N$ probabilities of the input belonging to each of the $N$ classes, all adding up to 1. In this kind of scenario, the decision rule usually is simply picking the highest probability class.

In the cases where labels are not mutually exclusive, each label's threshold can be individually selected just like in the binary classification case.

$\endgroup$
3
  • $\begingroup$ What do you mean by "In the cases where labels are not mutually exclusive". Supervised Classification problem means that the labels are mutually exclusive.... am I missing good to learn content ? If you're correct, could you pls, provide a link to an example $\endgroup$ Jun 30 at 14:37
  • $\begingroup$ There is a task called multi-label classification where there can be many labels associated, therefore not being mutually exclusive. Paperswithcode describes multiple benchmark datasets of such a type. $\endgroup$
    – noe
    Jul 1 at 8:01
  • $\begingroup$ Now I understand , thank you. I was aware of that multi-label classification, but it was unclear to me in the beginning what you meant. tnks $\endgroup$ Jul 1 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.