I am using OneVsRest Classifier in sklearn. So a multilabel model, 4 models for each class (i have 4 classes). When i called the predict_proba method i therefore get an array with 4 columns each one corresponding to a probability for that class. e.g.

0     1     2     3 
0.6  0.2   0.1   0.1 
0.8  0.05  0.05  0.1 

I know the models all train independently of one another and that the class asigned i.e. whether 0 1 2 3 takes the argmax of the 4 . what else happens under the hood with multilabel classifation such that each row sums up to 1? Why and how is this normalization happening.

  • $\begingroup$ Did you have a look on the ground based paper from by Trevor Hastie and Robert Tibshirani in 1998? projecteuclid.org/journals/annals-of-statistics/volume-26/… $\endgroup$ Sep 17 at 14:15
  • $\begingroup$ No i have not seen this $\endgroup$
    – Maths12
    Sep 17 at 14:24
  • 1
    $\begingroup$ Don't confuse multiclass classification (every instance has exactly one class) with multi-label classification (an instance can have any number of classes, from 0 to all of them). One-vs-rest is the standard method used for multiclass classification, it's not relevant for multi-label. Apparently you want to use multi-label but you're currently using multiclass. $\endgroup$
    – Erwan
    Sep 17 at 14:33
  • 1
    $\begingroup$ I was also going to point out the difference between multi-label classification. But @Erwan, if the sklearn probabilities sum to 1, OP is probably actually doing a multiclass classification. $\endgroup$ Sep 17 at 14:37
  • 1
    $\begingroup$ @Maths12 I see the confusion, I will try to explain this point in an answer. $\endgroup$
    – Erwan
    Sep 17 at 22:18

In multiclass classification, the assumption is that every instance has exactly one class. Example: a poll asks people their favourite colour among blue (B), yellow (Y) or red (R). Each instance represents a person's answer, either B, Y or R. The "one vs. rest" method means that 3 binary classifiers are trained:

  • "B" vs "not B", where the Y and R instances are labelled "not B"
  • "Y" vs "not Y", where the B and R instances are labelled "not Y"
  • "R" vs "not R", where the B and Y instances are labelled "not R"

These models are not independent by assumption, for example:

  • if the class is B then it cannot be Y or R.
  • if the class is not Y then it's either B or R.
  • Etc.

In probabilistic terms this translates as a distribution which sums to 1, because if a class has a high probability then it's impossible that any other class also has a high probability (complement). This is why the probabilities predicted by the binary classifiers are each divided by the sum (see Ben's answer for details).

Note: by contrast multi-label classification allows every instance to have any number of classes. In the example above it's as if the poll asks people to say whether they like each colour B, Y, R. A person might like all 3 colours or none of them. This implies that the binary classifiers are independent:

  • For "B vs not B", both the B and "not B" classes can contain instances which also have Y or R (or both).
  • As a consequence the classifiers are independent: knowing that an instance has class B doesn't imply anything about the other classes.

The normalization is straightforward division by the sum of the probabilities: source.

As to why, it's obviously desirable to have a sum of 1, but beyond that it's perhaps subjective. There's a discussion here suggesting that simple normalization outperformed other aggregations, but if I'm reading the linked paper correctly, it was a marginal gain and inconsistent across other calibration methods and datasets. I also like the answer here, which emphasizes that for any of this to make sense you need reasonably well-calibrated individual model predictions, and in that case the sum should be close to 1 already.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.