Do I need to correct predict_proba by training fraction?

Question

Many algorithms provide a predict_proba function indicating probability of a case to belong to that class (e.g. https://scikit-learn.org/stable/modules/generated/sklearn.svm.libsvm.predict_proba.html ).

Quoting from the answer by @Media at Explain Binary Classification with output 0.5 (True)

Suppose that you have a car classifier for distinguishing between white and blue cars. during training you had 100 images of blue car and 20 images of white car. During recall phase, if for an arbitrary image you have 50 percent for each class...

If blue cars accounted for 83% of training cases, and I get predict_proba for a car to be blue to be 0.5, do I take the probability to be 0.5 or do I need to correct it by a factor of 0.83?

If I do need to correct, do I multiply the factor (0.5*0.83) or divide it (0.5/0.83) to get the correct probability?

Answer 1

If blue cars accounted for 83% of training cases, and I get predict_proba for a car to be blue to be 0.5, do I take the probability to be 0.5 or do I need to correct it by a factor of 0.83?

First of all, the fact that your training dataset consists of 83% blue cars is not the same thing as the probability of the label being blue being 83%. This is the case only if the classifier is calibrated. sklearn team actually document the uncalibrated performance of their popular classifiers:

They state that:

Well calibrated classifiers are probabilistic classifiers for which the output of the predict_proba method can be directly interpreted as a confidence level. For instance a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to 0.8, approx. 80% actually belong to the positive class.

If you would like to be able to interpret the probability of the label as a confidence interval, you should calibrate your classifier using CalibratedClassifierCV. You can do that after you have trained your classifier, by setting cv=prefit in CalibratedClassifierCV.

Finally, if you are dealing with unbalanced classes and you want to take that into account during training, make sure to exploit the parameter class_weight, so that under-represented classes have a higher weight during training.

Answer 2

It depends on the model, whether your training data is representative of the testing data, and possibly on how "easy" the classification problem is. But the short answer is that (probably) no adjustment is necessary.

The model is exposed to the imbalance in the training data; most models will have that baked into their probability scores, so to trust the given probabilities requires that the testing data have a similar distribution. This goes to what exactly we mean by "the probability" that the new car is blue; I think in most cases, you would want the training data to be representative of the data in your use-case, so that we really do want the probability to be aware of the imbalance. [Now, sometimes in the presence of extreme imbalance, certain models might learn better with some adjustment, in which case you'll want also some "probability calibration" post-processing, but that's generally more complex than the simple multiply-or-divide method you suggest. In logistic regression for example, there's a well-known additive adjustment to the constant term, so that turns into a multiplicative adjustment to the log-odds, which does not translate into such a nice transformation on probabilities.]

Finally, some models implicitly skew their scores, so predict_proba should be taken with a fair bit of skepticism. Again, look up "probability calibration."