I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities from the models do not match the actual probabilities in the modeling data.

Is there a way to adjust the predicted probabilities from the class weighted models to match the actual probabilities in the data? I have seen equations for under-sampling (https://www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf) but they don't seem to work for class weights. I have searched online for an answer but maybe I am not using the right language?

Thank you!

  • 1
    $\begingroup$ This reference is exactly what you're looking for: gking.harvard.edu/files/0s.pdf Logistic Regression in Rare Events Data (Gary King) $\endgroup$ Jul 12 '20 at 18:13
  • $\begingroup$ Link to ssci2015_calibrating.pdf is dead today. Can you find online? I cannot because you did not give an author's name or a title of the article. $\endgroup$
    – pauljohn32
    Mar 9 '21 at 23:57
  • $\begingroup$ @pauljohn32, I believe it was "Calibrating Probability with Undersampling for Unbalanced Classification" by Dal Pozzolo et al. $\endgroup$ Mar 10 '21 at 1:51

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists:

Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where $p_1$ is the proportion of the positive class in the original dataset, and $r_1$ is the proportion of the positive class in the resampled dataset.

Equivalently, multiply the odds by the quantity inside the logarithm. (Unfortunately, this doesn't lead to a clean adjustment directly to the probabilities.)

Let's do a little rewriting to see the connection to your linked paper. $1-r_1$ is the proportion of negative classes call it $r_0$, and similarly with $p_1$. Use capitals $R_1, \dotsc$ to denote the number (or total weight) of samples rather than the proportions, and without subscripts $P,R$ to denote total numbers (or weights) of samples before and after resampling. So the multiplier becomes $$\frac{p_1(1-r_1)}{(1-p_1)r_1} = \frac{p_1 r_0}{p_0 r_1} = \frac{(P_1/P) (R_0/R)}{(P_0/P) (R_1/R)} = \frac{P_1 R_0}{P_0 R_1}.$$ In the context of the linked paper, positive class samples are not resampled, so $P_1=R_1$ and the adjustment simplifies to $R_0/P_0$, which is the parameter $\beta$ used in the paper.

Finally, using their equation (4), we check the change in odds: $$\text{new odds} = \frac{p}{1-p} = \frac{1}{\frac1p - 1} = \frac{1}{\frac{\beta p_s−p_s+ 1}{\beta p_s} - 1} = \frac{\beta p_s}{1-p_s} = \beta\cdot\text{old odds}. $$

So, what about weightings instead of resampling? Well, class_weights might have different effects in different algorithms, but generally the idea is that (positive) integer values of class_weights should correspond to duplicating the samples that many times, and fractional values interpolate that. So, it should be about the same to use the multiplicative factor above. Using the size version rather than the proportion version, we should interpret $R_0$ and $R_1$ as the total weights of the relevant classes.

I've not been able to find a reference for this version, so I put together a short experiment; it seems to verify that this shift works.
GitHub/Colab notebook

Finally, this shift in log-odds will fail to produce properly calibrated probabilities if the classifier is poorly calibrated on the weighted data. You could look into calibration techniques, from Platt to Beta to Isotonic. In this case, the shift above is probably superfluous.

  • $\begingroup$ Wow. Thanks so much. I haven't had a chance to look at the GitHub notebook yet, I have been out of town. Thank you for providing such a thoughtful answer. $\endgroup$ Sep 14 '19 at 2:41
  • $\begingroup$ Sorry I took so long to accept the question as answered. I had a close look at your code, which took some time, and it seems to work. The only thing is that you left r1 out of your adjustment, which assumes that r1 = 0.5. This works for the class weight scheme you used but if someone uses a different weighting scheme, r1 should be included. Thanks again! $\endgroup$ Sep 18 '19 at 17:27

The difference between predicted probabilities and actual probabilities is called training error.

There are many ways to reduce training error. Engineering better features and choosing a different machine learning algorithm are the most common.

  • 1
    $\begingroup$ Sorry, that is not quite what I am after. Without using class weights, the model will predict the actual probabilities. But when using class weights, the model will not predict the actual probabilities. I am hoping to use class weights, and adjust/correct the predicted probabilities to match the actual probabilities. $\endgroup$ Sep 9 '19 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.